Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. Can you describe how Datorios is used in a day-to-day workflow for someone building streaming applications on Flink?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. When performing research and building prototypes of the projects, what is your process for incorporating user experience into the implementation of the product?

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects.

Project 147

Project Data Lake High Quality Data SQL 147

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. If you've learned something or tried out a project from the show then tell us about it!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. What are the core problems that you were addressing with this project?

Building 147

Building Machine Learning Metadata Data Integration 147

dbt Developer Hub

APRIL 19, 2023

Close alignment with actual business processes : Business processes and metrics are modeled and calculated as part of dimensional modeling. Part 1: Setup dbt project and database ​ Step 1: Install project dependencies ​ Before you can get started: You must have either DuckDB or PostgreSQL installed.

Building 145

Building PostgreSQL BI Database 145

Data Engineering Podcast

MARCH 31, 2024

Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. Dagster offers a new approach to building and running data platforms and data pipelines.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What was your process for defining the problem you are focused on solving and the method to achieve that outcome? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Knowledge Hut

APRIL 29, 2024

In project management, project scheduling encompasses listing activities, defining milestones and scheduling deliverables for delivery. This indicates that every project schedule must include a planned start date and planned finish date, estimated resources assigned to each activity and estimated duration of each activity.

Project 98

Project Management Certification Algorithm 98

Engineering at Meta

DECEMBER 6, 2023

This is the most significant milestone yet for this project, which began in earnest after Mark Zuckerberg outlined his vision for it in 2019. Bringing E2EE to Messenger has been a complex process, with every feature and product goal revealing further challenges that required careful consideration.

Building 145

Building Designing Consulting Accessible 145

Data Engineering Podcast

JULY 23, 2023

Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Building 130

Building Machine Learning SQL Python 130

Knowledge Hut

JANUARY 28, 2024

The change control process is a crucial aspect of project management intended to manage and regulate changes made to the project plan, schedule, and budget. These change control process steps are planning, analyzing, approval, testing, implementing, and closing. What is Change Control in Project Management?

Process 98

Process Project Designing Pharmaceutical 98

Knowledge Hut

APRIL 12, 2024

Project management plays a significant role in the success of every organization. It ensures that the project is on track, aids in efficient management of resources, and also keeps the stakeholders know what is project and what's happening in it. What is Project Management Structures, and Why is it Important?

Project 98

Project Management Certification Designing 98

Knowledge Hut

MAY 6, 2024

The scope keeps changing as the project makes progress on the agreed timeline, so, the estimation becomes a challenging task for the product owner. Agile software estimation is used for realistic planning to implementing the project requirements as per commitments. Master the art of project management and achieve new heights.

Project 98

Project Programming Certification Database 98

Lyft Engineering

JUNE 28, 2023

While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort. On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. We called this the Open Beta phase of the project.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Knowledge Hut

APRIL 30, 2024

Management is paramount to making any project successful, irrespective of the industry and the work scope. Moreover, there are varied aspects to manage to ensure successful project delivery. These experts follow well-established principles of project management and efficiently manage the complexities to deliver expected end products.

Project 98

Project Management Certification Programming 98

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Code and raw data repository:   Version control: GitHub Heavily using GitHub Actions for things like getting warehouse data from vendor APIs, starting cloud servers, running benchmarks, processing results, and cleaning up after tuns. Tech stack.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Knowledge Hut

MAY 9, 2024

The Lean Six Sigma Green Belt certification is an important step in becoming a master of the lean six sigma technique and leading improvement projects for a company. Before we jump to the six sigma green belt project examples and ideas, let us learn some of the basic aspects of Six Sigma Green Belt. It should not need help.

Project 98

Project Manufacturing Certification Consulting 98

Knowledge Hut

OCTOBER 29, 2023

Choosing the best computer science project topic is critical to the success of any computer science student or employee. After all, the more engaging and interesting topic, the more likely it is that students or employees will be able to stay motivated and focused throughout the duration of the project.

Computer Science 111

Computer Science Coding Project Hospitality 111

Knowledge Hut

APRIL 26, 2024

The product development process is just as vital as product management; both seem similar but have subtle variances. Product development focuses on the creation of a product, whereas The entire process is overseen by product management. What Is the Product Development Process? It involves seven product development process steps.

Process 98

Process Manufacturing Retail Electronics 98

Knowledge Hut

APRIL 16, 2024

Over several decades, projects have been initiated or undertaken due to market demands, business needs, at the behest of customer request, technological advancements and to comply with regulatory requirements. We will look at some of the key project management methods followed in today’s world. Know more about the project description.

Project 98

Project Management Programming Designing 98

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform.   We didn’t build our applications in neat containers, but in bulky monoliths which commingled business, database, backend, and frontend logic. Our deployments were initially manual. Apache started to log like a maniac.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

AUGUST 27, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold You shouldn't have to throw away the database to build with fast-changing data.

Database 130

Database Building PostgreSQL BI 130

The Pragmatic Engineer

AUGUST 22, 2023

Q: “I play video games, and I was wondering how building games compares to more ‘standard’ software development. At Skype, I worked on the team building Skype for the Xbox One console, and several engineers on that team had previously worked in games. To get the full issues, twice a week, subscribe here.

Software Engineer 207

Software Engineer Software Engineering Consulting Entertainment 207

Knowledge Hut

APRIL 16, 2024

Over the years I have found that my most popular blog posts are those that speak to entry-level project managers. Project management is a vast practice area and for somebody who has only recently started managing projects , it can seem overwhelming. That is true for all projects irrespective of the type of methodology they use.

Project 97

Project Management Certification Process 97

Knowledge Hut

MAY 6, 2024

Release Planning / Goal Plan & Prioritize your Backlog (Define User Stories like Taking a Course, Complete Application, Apply & Book for the exam) Set the release goal: “As a Project Management Professional, I want to Obtain X-Certification in order to gain more knowledge in Y-Industry.” Validate your Experience.

Certification 97

Certification Project Management Technology 97

Data Engineering Podcast

JANUARY 29, 2023

Data analysts create reports that are used by the business to understand and direct the business, but the process is very labor and time intensive. The team at Omni have taken a new approach by automatically building models based on the queries that are executed. Can you describe what Omni Analytics is and the story behind it?

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Knowledge Hut

APRIL 29, 2024

The biggest challenge is companies facing difficulties to keep their projects running during this pandemic, especially with how teams work and communicate. How can companies coordinate and work remotely, while also keeping their projects in sync? It can be challenging for certain project managers especially those who micro-manage.

Project 98

Project Portfolio Cloud Data Storage 98

Data Engineering Podcast

JANUARY 15, 2023

What was your process for identifying the "must have" vs "nice to have" features for launching the platform? If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!

Coding 130

Coding Building PostgreSQL Data Lake 130

Knowledge Hut

APRIL 29, 2024

P roject Communication Plan is an important step in Project Management Plan or what is simply called Project Plan. For more details on Project Management please refer to PMP certification course. What is Project Communication Plan?

Project 98

Project Consulting BI Algorithm 98

Knowledge Hut

APRIL 16, 2024

Going for CSM certification training and knowing how to build a self-organizing team as a Scrum master will help you get trained well. Most Agile-Scrum organizations emphasize on building the self-organizing team - why? How Do You Build a Self-Organizing Team as a Scrum Master?

Building 98

Building Certification Project Management 98

Data Engineering Podcast

DECEMBER 24, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you've learned something or tried out a project from the show then tell us about it! Data lakes are notoriously complex.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

OCTOBER 29, 2023

In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data projects are notoriously complex.

Database 189

Database BI SQL Machine Learning 189