Cloudera

SEPTEMBER 26, 2023

IBM and Cloudera’s common goal is to accelerate data-driven decision making for enterprise customers, working on defining and executing the best solution for each customer. You can now elevate your data potential and activate AI’s capabilities through the synergic integration between IBM watsonx and Cloudera.

Kafka 95

Kafka Technology IT Government 95

Christophe Blefari

MARCH 15, 2024

Understand how BigQuery inserts, deletes and updates — Once again Vu took time to deep dive into BigQuery internal, this time to explain how data management is done. Pandera, a data validation library for dataframes, now supports Polars. Unlocking Kafka's potential: tackling tail latency with eBPF.

Metadata 272

Metadata Data Software Engineer Datasets 272

Data Engineering Podcast

MAY 21, 2023

In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache. Can you describe what Estuary is and the story behind it? Stream processing technologies have been around for around a decade.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

Cloudera

OCTOBER 12, 2020

The DataFlow platform has established a leading position in the data streaming market by unlocking the combined value and synergies of Apache NiFi, Apache Kafka and Apache Flink. We recently delivered all three of these streaming capabilities as cloud services through Cloudera Data Platform (CDP) Data Hub on AWS and Azure.

Cloud 132

Cloud Process Scala Kafka 132

Data Engineering Weekly

MAY 4, 2025

link] Meta: How Meta understands data at scale Meta describes its data management practices as adopting a “shift-left” approach, integrating data schematization and annotations early in product development. TIL about the idle stream problem.

Data Engineering 67

Data Engineering Data Engineer Engineering Java 67

Data Engineering Podcast

OCTOBER 15, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Process 182

Process Building SQL BI 182

Data Engineering Podcast

FEBRUARY 3, 2018

Summary One of the critical components for modern data infrastructure is a scalable and reliable messaging system. Publish-subscribe systems have been popular for many years, and recently stream oriented systems such as Kafka have been rising in prominence.

Kafka 100

Kafka Data Pipeline Data Science Data Engineer 100

Data Engineering Podcast

FEBRUARY 3, 2018

Summary One of the critical components for modern data infrastructure is a scalable and reliable messaging system. Publish-subscribe systems have been popular for many years, and recently stream oriented systems such as Kafka have been rising in prominence.

Kafka 100

Kafka Data Pipeline Data Science Data Engineer 100

Data Engineering Podcast

SEPTEMBER 28, 2020

Summary Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine.

Kafka 100

Kafka Big Data BI Data Engineer 100

Confluent

MAY 22, 2019

In light of this, we’ll share an emerging machine-to-machine (M2M) architecture pattern in which MQTT, Apache Kafka ® , and Scylla all work together to provide an end-to-end IoT solution. The explosive number of devices generating, tracking and sharing data across a variety of networks is overwhelming to most data management solutions.

Kafka 98

Kafka Google Cloud NoSQL Entertainment 98

Data Engineering Podcast

DECEMBER 10, 2017

When you are processing that data in multiple systems it can be difficult to ensure that they all have an accurate representation of that schema, which is why Confluent has built a schema registry that plugs into Kafka. Conversely, what would be involved in using a storage backend other than Kafka?

Kafka 100

Kafka Data Pipeline Data Engineer Data Engineering 100

Data Engineering Podcast

MAY 11, 2020

Pulsar is a well engineered and robust platform for building the core of any system that relies on durable access to easily scalable streams of data. What is Pulsar’s role in the lifecycle of data and where does it fit in the overall ecosystem of data tools? Why is streaming data such an important capability?

Cloud 100

Cloud Lambda Architecture Kafka Hadoop 100

Data Engineering Podcast

JULY 6, 2020

Observability and governance of streaming data requires a different approach than batch oriented workflows, and this episode does an excellent job of outlining the complexities involved and how to address them. If you hand a book to a new data engineer, what wisdom would you add to it? Redis and Pulsar)?

Systems 100

Systems Kafka SQL Data Engineer 100

Cloudera

OCTOBER 7, 2021

The concept of the data mesh architecture is not entirely new; Its conceptual origins are rooted in the microservices architecture, its design principles (i.e., need to integrate multiple “point solutions” used in a data ecosystem) and organization reasons (e.g., The Value Proposition of CDF in Data Mesh Implementations.

Architecture 124

Architecture Metadata Kafka Government 124

Data Engineering Podcast

JANUARY 5, 2020

If you have ever struggled with implementing your own change data capture pipeline, or understanding when it would be useful then this episode is for you. You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.

Database 100

Database Kafka PostgreSQL MySQL 100

Edureka

APRIL 22, 2025

It lets you describe data more complexly and make predictions. AI-powered data engineering solutions make it easier to streamline the data management process, which helps businesses find useful insights with little to no manual work. This will help make better analytics predictions and improve data management.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

Cloudera

JANUARY 20, 2021

In order to enable connected manufacturing and emerging IoT use cases, ECC needs a solution that can handle all types of diverse data structures and schemas from the edge, normalize the data, and then share it with any type of data consumer including Big Data applications. . STEP 4: Capture data from Apache Kafka streams.

Manufacturing 110

Manufacturing Data Warehouse Kafka Retail 110

Lyft Engineering

NOVEMBER 29, 2023

Druid Data Ingestion Our pipeline for the two methods of ingesting data into Druid—the upper process is for batch ingestion, the lower process is for real-time ingestion. Then, they needed to define an ingestion specification which tells Druid how to process the data being ingested. This was our main form of ingestion.

Kafka 109

Kafka Data Ingestion Architecture Datasets 109

Data Engineering Podcast

APRIL 1, 2018

In this episode ThreatStack’s director of operations, Pete Cheslock, and senior infrastructure security engineer, Patrick Cable, discuss the data infrastructure that supports their platform, how they capture and process the data from client systems, and how that information can be used to keep your systems safe from attackers.

Amazon Web Services 100

Amazon Web Services Cloud PostgreSQL Kafka 100

Data Engineering Podcast

MAY 27, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. What is Alooma and what is the origin story? How is the Alooma platform architected?

Data Pipeline 100

Data Pipeline MongoDB Google Cloud Scala 100

Data Engineering Podcast

NOVEMBER 18, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. What are some of the primary ways that Flink is used?

Process 100

Process Google Cloud Scala Kafka 100

Data Engineering Podcast

DECEMBER 2, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode.

Systems 100

Systems Building Kafka Java 100

Data Engineering Podcast

JULY 8, 2018

They also explained how it fits in the broad landscape of data tools, the interesting and challenging aspects of the project, and how to build new extensions. Where does it sit in the broader landscape of data tools? How do you manage versioning and backup of data flows, as well as promoting them between environments?

Building 100

Building Transportation Kafka Java 100

Data Engineering Podcast

DECEMBER 9, 2018

He also discusses what you need to know to get it deployed and keep it running in a production environment and how it fits into the overall data ecosystem. How does it compare to some of the other streaming frameworks such as Flink, Kafka, or Storm? Can you start by explaining what Spark is? Who uses Spark? Who uses Spark?

MySQL 100

MySQL Scala Kafka Hadoop 100

Data Engineering Podcast

DECEMBER 31, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. How do you represent a stream on-disk?

Lambda Architecture 100

Lambda Architecture Process Data Process Kafka 100

Data Engineering Podcast

JULY 27, 2021

This is a fascinating conversation with a lot of useful lessons for anyone who wants to understand the operational aspects of Pulsar and the benefits that it can provide to data workloads. How do you handle migrating existing projects, particularly if they are using Kafka currently?

Management 100

Management Building Kafka Data Warehouse 100

Data Engineering Podcast

MAY 26, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Can you start by sharing some of your experiences with data migration projects? Can you start by sharing some of your experiences with data migration projects?

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

AUGUST 6, 2017

Interview Introduction How did you first get involved in the area of data management? Regulatory challenges of processing other people’s data What does your data pipelining architecture look like? What are the most challenging aspects of building a general purpose data management environment?

PostgreSQL 100

PostgreSQL MongoDB Data Pipeline Kafka 100

Data Engineering Podcast

AUGUST 3, 2021

In this episode Vinoth shares the history of the project, how its architecture allows for building more frequently updated analytical queries, and the work being done to add a more polished experience to the data lake paradigm. Interview Introduction How did you get involved in the area of data management?

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

OCTOBER 5, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Engineering 100

Engineering MongoDB Kafka Data Lake 100

Confluent

AUGUST 31, 2023

Learn how Confluent’s cloud-native Apache Kafka engine stood out from other data management systems with its uniquely elastic, reliable, and cost-efficient design

Kafka 89

Kafka Cloud Designing Data Management 89

Data Engineering Podcast

MAY 20, 2018

In this episode he describes how Presto is architected, how you can use it for your analytics, and the work that he is doing at Starburst Data. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform.

PostgreSQL 100

PostgreSQL Hadoop SQL Kafka 100

Data Engineering Podcast

MAY 18, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

NOVEMBER 11, 2018

In this episode Yoni Iny, CTO of Upsolver, discusses the various components that are necessary for a successful data lake project, how the Upsolver platform is architected, and how modern data lakes can benefit your organization. There are a lot of opinions on both sides of the data lake argument.

Data Lake 100

Data Lake Building Kafka Cloud 100

Data Engineering Podcast

OCTOBER 5, 2020

In this episode founders DeVaris Brown and Ali Hamidi explain how their tenure at Heroku informed their approach to making data integration self service, how the platform is architected, and how they have designed their system to adapt to the continued evolution of the data ecosystem. What do you have planned for the future?

Data Integration 100

Data Integration Kafka Data Lake Machine Learning 100

Data Engineering Podcast

MARCH 8, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

IT 100

IT Data Warehouse MongoDB Kafka 100