Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. When performing research and building prototypes of the projects, what is your process for incorporating user experience into the implementation of the product? Closing Announcements Thank you for listening!

Data Process 147

Data Process Process Data Lake High Quality Data 147

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

The Pragmatic Engineer

APRIL 21, 2024

Sometimes the best explanations of how a technology solution works come from the software engineers who built it. A refresher on OpenAI, and on Evan Evan: how did you join OpenAI, and end up heading the Applied engineering group – which also builds ChatGPT? We do it through a process called  pretraining. and GPT-4.

Engineering 351

Engineering Software Engineer Software Engineering Programming 351

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. We feel your pain. It ends up being anything but that.

Building 147

Building Machine Learning Metadata Data Integration 147

The Pragmatic Engineer

JUNE 1, 2023

Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js Change deploys to be a one-command process, instead of multiple steps See blog posts #1-11 for details on all the steps. ” stage Improve deploys.

Education 363

Education Project PostgreSQL Software Engineer 363

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What was your process for defining the problem you are focused on solving and the method to achieve that outcome? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

JULY 23, 2023

Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Building 130

Building Machine Learning SQL Python 130

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform. Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company. Joshua has remained technical while working as an executive. Subscribe here. Our deployments were initially manual.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Engineering at Meta

DECEMBER 6, 2023

Bringing E2EE to Messenger has been a complex process, with every feature and product goal revealing further challenges that required careful consideration. End-to-end encryption isn’t about the technology at its core. Some of these just had to be rebuilt from scratch, while others required deploying new applied privacy technology.

Building 145

Building Designing Consulting Accessible 145

databricks

NOVEMBER 26, 2024

Large language models are revolutionizing how we interact with technology by leveraging advanced natural language processing to perform complex tasks. In recent years.

Building 116

Building Technology Process Machine Learning 116

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Monte Carlo

JANUARY 3, 2024

But building a generative AI model that actually drives business value is hard. Sure, you could integrate your AI model more deeply into your organization’s processes, but that feels risky. Snowflake Cortex now enables organizations to quickly analyze data and build AI apps directly in Snowflake.

Technology 113

Technology Database Data Engineering Data Engineer 113

Christophe Blefari

MARCH 15, 2024

AI News 🤖 Mira Murati answers the Wall Street Journal about OpenAI Sora — OpenAI CTO has been asked a few questions about the underlying technology in Sora. The technology under this, is, Cityvision. Building Meta’s GenAI infrastructure — 2x 24k GPU clusters and it's growing.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Data Engineering Podcast

AUGUST 27, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold You shouldn't have to throw away the database to build with fast-changing data.

Database 130

Database Building PostgreSQL BI 130

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

JANUARY 29, 2023

Data analysts create reports that are used by the business to understand and direct the business, but the process is very labor and time intensive. The team at Omni have taken a new approach by automatically building models based on the queries that are executed. Can you describe what Omni Analytics is and the story behind it?

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Knowledge Hut

APRIL 26, 2024

The product development process is just as vital as product management; both seem similar but have subtle variances. Product development focuses on the creation of a product, whereas The entire process is overseen by product management. What Is the Product Development Process? It involves seven product development process steps.

Process 98

Process Manufacturing Retail Electronics 98

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe a typical workflow for teams that are building on top of RisingWave?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment.

BI 208

BI SQL Data Machine Learning 208

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” And COBOL was just one of many attempts.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

Snowflake

NOVEMBER 1, 2023

Snowflake has invested heavily in extending the Data Cloud to AI/ML workloads, starting in 2021 with the introduction of Snowpark , the set of libraries and runtimes in Snowflake that securely deploy and process Python and other popular programming languages.

Building 111

Building Python SQL Programming Language 111

Data Engineering Podcast

MARCH 3, 2024

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Your first 30 days are free!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

OCTOBER 29, 2023

In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection. When a team is selecting a database technology for a given task, what are the types of questions that they should be asking?

Database 189

Database BI SQL Machine Learning 189

Data Engineering Podcast

MAY 21, 2023

Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. In Code Comments, host Jamie Parker, Red Hatter and experienced engineer, shares the journey of technologists from across the industry and their hard-won lessons in implementing new technologies. My thanks to the team at Code Comments for their support.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Snowflake

JUNE 5, 2024

Enterprises can utilize gen AI to extract more value from their data and build conversational interfaces for customer and employee applications. Addressing a lack of in-house AI expertise and simplifying AI processes can make adoption easier. The embedding process is completely automated for easy configuration.

Coding 93

Coding Building Management Government 93

The Pragmatic Engineer

AUGUST 22, 2023

Q: “I play video games, and I was wondering how building games compares to more ‘standard’ software development. At Skype, I worked on the team building Skype for the Xbox One console, and several engineers on that team had previously worked in games. To get the full issues, twice a week, subscribe here.

Software Engineer 207

Software Engineer Software Engineering Consulting Entertainment 207

Christophe Blefari

MARCH 1, 2023

Also dbt only does a pass-through to your underlying data compute technology, there is not any kind of processing within dbt. There are multiple technological reasons for this, but technology is rarely the real reason. Actually dbt can be seen as an orchestrator with no scheduling capabilities.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Data Engineering Podcast

FEBRUARY 11, 2024

In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

JUNE 30, 2024

Petr emphasizes the need for a holistic approach to data reliability, integrating data systems into broader business processes. For a team who is onboarding onto Synq, what are the steps required to get it integrated into their technology stack and workflows? What does a typical incident/error resolution process look like with Synq?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Rockset

JUNE 21, 2024

We are thrilled to join the OpenAI team and bring our technology and expertise to building safe and beneficial AGI. We will gradually transition current customers off Rockset and are committed to ensuring a smooth process. I’m excited to share that OpenAI has completed the acquisition of Rockset.

Database 145

Database Cloud Accessible Accessibility 145

Data Engineering Podcast

SEPTEMBER 24, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold You shouldn't have to throw away the database to build with fast-changing data.

SQL 147

SQL BI Machine Learning Python 147

Data Engineering Podcast

OCTOBER 8, 2023

Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Insurance 162

Insurance BI SQL Machine Learning 162