Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Go to dataengineeringpodcast.com/dagster today to get started.

Database 162

Database Technology Data Lake High Quality Data 162

Engineering at Meta

MARCH 4, 2025

Shane sits down with Pascal Hartig to share how his team is building foundational models for the Ray-Ban Meta glasses. They talk about the unique challenges of AI glasses and pushing the boundaries of AI-driven wearable technology.

Building 81

Building Programming Engineering Technology 81

Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162

Building BI SQL Machine Learning 162

Robinhood

AUGUST 12, 2024

We are thrilled to announce that Jeff Pinner has joined Robinhood as Chief Technology Officer (CTO). Great engineering is critical to our ability to build cutting-edge financial products at Robinhood, which is why we’re excited to welcome Jeff as CTO,” said Vlad Tenev, CEO and Co-Founder of Robinhood.

Technology 116

Technology Finance Engineering Building 116

Precisely

MAY 5, 2025

However, theres often a debate on whether to build a custom in-house solution or purchase an enterprise-grade platform. And I get it on the surface, building often seems like it might be the less expensive option, especially these days when cloud vendors offer tempting incentives and the tools seem more accessible than ever.

Building 58

Building Telecommunication Certification Data Security 58

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Scott Logic

JANUARY 27, 2025

In this episode, Im joined by Technology Lead Andrew Carr and CTO Colin Eberhardt to delve into the evolving nature of technology strategies within organisations. As technological advancements accelerate, we question the relevance of a traditional long-term technology strategy and whether it has become an industry buzzword in itself.

Technology 52

Technology Building IT 52

Precisely

JANUARY 9, 2025

Disclaimer: Throughout this post, I discuss a variety of complex technologies but avoid trying to explain how these technologies work. The goal of this post is to understand how data integrity best practices have been embraced time and time again, no matter the technology underpinning. Then came Big Data and Hadoop!

Data Integration 118

Data Integration Hadoop Data Warehouse Data Lake 118

Data Engineering Weekly

DECEMBER 28, 2024

Enterprises are encouraged to experiment with AI, build numerous small-scale agents, learn from each, and expand their agent infrastructure over time. These platforms are instrumental in building the robust data infrastructure necessary to support the burgeoning field of AI agents.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. To address this limitation, we utilize Privacy Probes , a key component of our PAI lineage technologies. helping inform the right places to apply privacy controls.

Data Warehouse 120

Data Warehouse SQL Programming Language Data 120

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Engineering at Meta

DECEMBER 6, 2023

End-to-end encryption isn’t about the technology at its core. High-level approach With all of this in mind, our high-level approach was to build off of Meta’s prior learnings in E2EE, from both WhatsApp and Messenger’s Secret Conversations, and then to iterate on our most challenging problems.

Building 145

Building Designing Consulting Accessibility 145

Data Engineering Podcast

JANUARY 28, 2024

For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics. When is Scanner the wrong choice?

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

MAY 5, 2024

In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building 147

Building Data Lake High Quality Data Machine Learning 147

databricks

JULY 23, 2024

The communications industry is experiencing immense change due to rapid technological advancements and evolving market trends. Communications service providers (CSP) build various solutions.

Building 116

Building Technology 116

Snowflake

DECEMBER 18, 2023

Data clean rooms have emerged as the technology to meet this need, enabling interoperability where multiple parties can collaborate on and analyze sensitive data in a governed way without exposing direct access to the underlying data and business logic. Snowflake’s acquisition of Samooha is subject to customary closing conditions.

Cloud 136

Cloud Building Entertainment Government 136

The Pragmatic Engineer

JUNE 1, 2023

Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js Phase 2: some business logic, and more infra (December-January) Draw a map using JavaScript to map onto an SVG format Build a graph and traverse it.

Education 364

Education Project PostgreSQL Software Engineer 364

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform. Joshua is currently VP of Product & Strategy at VMware, a cloud computing and virtualization technology company. This project’s advancements laid the groundwork for many technologies still with us today, seven decades later.

Engineering 337

Engineering Bytes Cloud Computing AWS 337

The Pragmatic Engineer

APRIL 21, 2024

Sometimes the best explanations of how a technology solution works come from the software engineers who built it. A refresher on OpenAI, and on Evan Evan: how did you join OpenAI, and end up heading the Applied engineering group – which also builds ChatGPT? "How does ChatGPT work, under the hood? My questions are in italic.

Engineering 363

Engineering Software Engineer Software Engineering Programming 363

Snowflake

JANUARY 21, 2025

The winners will be those that adopt forward-thinking data strategies, build trust with partners and clients, and leverage AI to deliver real-time insights and personalized campaigns. Agencies today can build or adopt platforms to deliver data-driven marketing strategies to brands. Art and science within agencies were often separated.

Media 94

Media Unstructured Data Data Retail 94

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). Technological innovation rarely happens in a vacuum; it builds on previous technologies.

Programming Language 305

Programming Language Project Coding Engineering 305

Snowflake

MARCH 31, 2025

Many enterprises are already using Container Runtime to cost-effectively build advanced ML use cases with easy access to GPUs. Customers include CHG Healthcare, Keysight Technologies and Avios. CHG builds and productionizes its end-to-end ML models in Snowflake ML. With over $5.5

Healthcare 90

Healthcare Medical Government Food 90

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. For someone who is interested in building a data lakehouse with Trino and Iceberg, how does that influence their selection of other platform elements?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Christophe Blefari

MARCH 15, 2024

AI News 🤖 Mira Murati answers the Wall Street Journal about OpenAI Sora — OpenAI CTO has been asked a few questions about the underlying technology in Sora. The technology under this, is, Cityvision. Building Meta’s GenAI infrastructure — 2x 24k GPU clusters and it's growing.

Metadata 272

Metadata Data Software Engineer Software Engineering 272

Snowflake

MARCH 19, 2025

Nearly nine out of 10 business leaders say their organizations data ecosystems are ready to build and deploy AI, according to a recent survey. Snowflake experts, customers and partners will share strategic insights and practical tips for building a solid and collaboration-ready data foundation for AI.

Retail 89

Retail Media Entertainment Unstructured Data 89

The Pragmatic Engineer

OCTOBER 10, 2023

He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. The focus seemed to shift to: invent something new → build a service for it → ship it.

Software Engineer 318

Software Engineer Software Engineering Architecture Media 318

databricks

NOVEMBER 26, 2024

Large language models are revolutionizing how we interact with technology by leveraging advanced natural language processing to perform complex tasks. In recent years.

Building 116

Building Technology Process Machine Learning 116

Snowflake

NOVEMBER 11, 2024

At Snowflake BUILD , we are introducing powerful new features designed to accelerate building and deploying generative AI applications on enterprise data, while helping you ensure trust and safety. Customers such as Avios, CHG Healthcare and Keysight Technologies are already developing container-based models in Snowflake ML.

Unstructured Data 96

Unstructured Data SQL AWS Healthcare 96

Data Engineering Podcast

APRIL 23, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What do you have planned for the future of Meroxa? Closing Announcements Thank you for listening! Don't forget to check out our other shows.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277