Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Data Engineering Podcast

MARCH 5, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. What do you have planned for the future of this topic at Data Council events?

Building 147

Building Machine Learning Database Design Metadata 147

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How have the recent breakthroughs in large language models (LLMs) improved your ability to build features in Zenlytic? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Data Engineering Podcast

APRIL 23, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Snowflake

NOVEMBER 15, 2023

BUILD 2023 is where AI gets real. Join our two-day virtual global conference and learn how to build with the app dev innovations you heard about at Snowflake Summit and Snowday. As always, BUILD is created by builders, for builders, so you’ll find plenty to spark your interest and boost your skill set.

Building 129

Building Accessible Accessibility IT 129

Pinterest Engineering

MARCH 26, 2025

Personalization Stack Building a Gift-Optimized Recommendation System The success of Holiday Finds hinges on our ability to surface the right gift ideas at the right time. Unified Logging System: We implemented comprehensive engagement tracking that helps us understand how users interact with gift content differently from standardPins.

Building 59

Building Engineering Algorithm Systems 59

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. In this episode Gleb Mezhanskiy shares some valuable advice and insights into how you can build reliable and well-tested data assets with dbt and data-diff. RudderStack also supports real-time use cases.

Project 130

Project Building Data Lake Machine Learning 130

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. The majority of products that are available either require too much effort to structure the logs, or aren't fast enough for interactive use cases.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Striim

APRIL 22, 2025

From delivering event-driven predictions to powering live recommendations and dynamic chatbot conversations, AI/ML initiatives depend on the continuous movement, transformation, and synchronization of diverse datasets across clouds, applications, and databases. Define the must-have characteristics of a data streaming architecture.

Architecture 52

Architecture Data Architecture Building Datasets 52

Lyft Engineering

JUNE 28, 2023

On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases. One key component is the Analytics Event Abstraction layer.

Machine Learning 122

Machine Learning Building Kafka Metadata 122

Snowflake

MARCH 19, 2025

Nearly nine out of 10 business leaders say their organizations data ecosystems are ready to build and deploy AI, according to a recent survey. Snowflake experts, customers and partners will share strategic insights and practical tips for building a solid and collaboration-ready data foundation for AI.

Retail 92

Retail Media Entertainment Unstructured Data 92

Engineering at Meta

JANUARY 21, 2025

For example, a profiler takes a sample every N events (or milliseconds in the case of time profilers) to understand where that event occurs or what is happening at the moment of that event. With a CPU-cycles event, for example, the profile will be CPU time spent in functions or function call stacks executing on the CPU.

Technology 116

Technology Metadata Utilities Engineering 116

dbt Developer Hub

APRIL 19, 2023

Each record in the fact table represents a business event such as a: Item sale Website click Production work order There are two tables in the sales schema that catch our attention. fct_sales table Dimension tables ​ info Dimension tables are used to represent contextual or descriptive information for a business process event.

Building 145

Building PostgreSQL BI Database 145

Cloudera

JANUARY 7, 2025

Airports are an interconnected system where one unforeseen event can tip the scale into chaos. Building a data culture Paul: You joined Halifax International Airport Authority over a year ago. Tell me about what you were trying to build or replace or accomplish. Ryan: First, I wanted to build a culture. Thats not it.

Finance 81

Finance Building Cloud Engineering 81

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Snowflake

APRIL 20, 2023

Snowflake users are already taking advantage of LLMs to build really cool apps with integrations to web-hosted LLM APIs using external functions , and using Streamlit as an interactive front end for LLM-powered apps such as AI plagiarism detection , AI assistant , and MathGPT. Join us in Vegas at our Summit to learn more.

Building 138

Building Unstructured Data Government Coding 138

Snowflake

NOVEMBER 4, 2024

Now, it’s time to BUILD. Join us for BUILD 2024, a three-day global virtual conference taking place Nov. 12-15, to hear major Snowflake product announcements firsthand and to learn how to build with our latest innovations through dozens of technical sessions and hands-on labs. Cost-effectiveness! Efficiency!)

Unstructured Data 105

Unstructured Data Python Machine Learning Data Pipeline 105

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 116

Cloud Building Insurance Data Engineering 116

Striim

FEBRUARY 4, 2025

This recipe shows how you can build a data pipeline to read data from Salesforce and write to BigQuery. Benefits Act in Real Time – Predict, automate, and react to business events as they happen, not minutes or hours later. Empower Your Teams – Give teams across your organization a real-time view into operational data.

Building 52

Building Data Pipeline Datasets Finance 52

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. ” Like most startups, Spare Cores also made their own “expensive mistake” while building the product: “We accidentally accumulated a $3,000 bill in 1.5 We envision building something comparable to AWS Fargate , or Google Cloud Run.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Engineering at Meta

FEBRUARY 13, 2025

We also celebrated the security research done by our bug bounty community as part of our annual bug bounty summit and many other industry events. As part of our defense-in-depth strategy , we continued to collaborate with the security research community in the areas of GenAI, AR/VR, ads tools, and more.

Programming 114

Programming Designing Accessibility Accessible 114

Confluent

JANUARY 14, 2025

PodPrep AI, an AI-powered research assistant, leverages EDA and real-time streaming data using Confluent and Flink, in order to help its author with podcast preparation.

Kafka 80

Kafka Building IT Data 80

Snowflake

DECEMBER 18, 2023

Samooha’s intuitive user interface makes it easy to build data clean rooms that run as Snowflake Native Apps in the Data Cloud. Building a data clean room from the ground up can be challenging and require technical expertise and significant investment. Snowflake’s acquisition of Samooha is subject to customary closing conditions.

Cloud 138

Cloud Building Entertainment Government 138

Snowflake

APRIL 2, 2025

The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems. Learn more Join Snowflake at Iceberg Summit , a two-day event taking place in San Francisco on April 8 and virtually April 9. Now you dont have to choose.

Data Lake 96

Data Lake Cloud Storage Metadata Data Warehouse 96

The Pragmatic Engineer

JUNE 1, 2023

Phase 2: some business logic, and more infra (December-January) Draw a map using JavaScript to map onto an SVG format Build a graph and traverse it. The project looks like a tough one to build from scratch on the side. Here’s a video of Juraj demonstrating this traversal. Incremental progress.

Education 364

Education Project PostgreSQL Software Engineer 364

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. Or, experience these features firsthand at our free Dev Day event on June 6th in the Demo Zone.

Data Pipeline 124

Data Pipeline Python Data Engineering Data Engineer 124

The Pragmatic Engineer

MARCH 3, 2023

I asked Googlers the reason why these events have been canceled and one thing became clear: most of the program managers who worked on the coding competitions were recently let go in Google’s historic job cuts. So knowing these lead up signs, the cancellation of the event is not that surprising, considering the info we had beforehand.

Coding 218

Coding IT Software Engineer Software Engineering 218

The Pragmatic Engineer

OCTOBER 31, 2023

” And an update inside the final hour and a half of the outage: “2:00 PM PDT Many AWS services are now fully recovered and marked Resolved on this event. At that time, we began processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services.

AWS 290

AWS Google Cloud Cloud Engineering 290

Cloudera

NOVEMBER 13, 2024

Build and test training and inference prompts. Fine Tuning Studio ships with powerful prompt templating features, so users can build and test the performance of different prompts to feed into different models and model adapters during training. Data Preparation.

Datasets 87

Datasets Machine Learning Coding Data Preparation 87

Cloudera

DECEMBER 4, 2024

delivers on this need, providing enhancements that streamline development, boost efficiency, and empower organizations to build cutting-edge GenAI solutions. These enhancements empower organizations to build sophisticated GenAI solutions with greater ease and efficiency, unlocking the transformative power of AI. Cloudera DataFlow 2.9

Data Pipeline 93

Data Pipeline Data Ingestion Data Preparation Architecture 93

Netflix Tech

FEBRUARY 14, 2025

We will explore the challenges we encounter and unveil how we are building a resilient solution that transforms these client-side impressions into a personalized content discovery experience for every Netflixviewer. These events are promptly relayed from the client side to our servers, entering a centralized event processing queue.

Kafka 66

Kafka Datasets Metadata Utilities 66

Data Engineering Weekly

MAY 4, 2025

deployment on Astro to test DAG versioning, backfills, event-driven scheduling, and more. Get started → Editor’s Note: OpenXData Conference - 2025 - A Free Virtual Event A free virtual event on open data architectures - Iceberg, Hudi, lakehouses, query engines, and more. Spin up a new 3.0

Data Engineering 67

Data Engineering Data Engineer Engineering Java 67