Data Engineering Podcast

APRIL 23, 2023

Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Team Data Science

MAY 10, 2020

Kafka, while not in the top 5 most in demand skills, was still the most requested buffer technology requested which makes it worthwhile to include it. I'll use Python and Spark because they are the top 2 requested skills in Toronto. The remaining tech (stages 3, 4, 7 and 8) are all AWS technologies.

Data Pipeline 130

Data Pipeline Data Ingestion AWS Kafka 130

Confluent

OCTOBER 4, 2022

Confluent’s new Stream Designer is the industry’s first visual interface for rapidly building, testing, and deploying streaming data pipelines natively on Apache Kafka.

Data Pipeline 105

Data Pipeline Designing Kafka Data 105

Picnic Engineering

FEBRUARY 8, 2023

In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. In this post, we will explain how our team automates the creation of new data pipeline deployments. Now, we can have a pipeline ready in minutes.

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Precisely

DECEMBER 28, 2023

Business success is based on how we use continuously changing data. That’s where streaming data pipelines come into play. This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. What is a streaming data pipeline?

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. Data stacks are becoming more and more complex.

Data Pipeline 130

Data Pipeline Building MongoDB MySQL 130

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®.

Kafka 95

Kafka Cloud Data Pipeline PostgreSQL 95

Confluent

JUNE 13, 2019

This data pipeline is a great example of a use case for Apache Kafka ®. The data processing pipeline characterizes these objects, deriving key parameters such as brightness, color, ellipticity, and coordinate location, and broadcasts this information in alert packets. The case for Apache Kafka.

Kafka 102

Kafka Python Bytes Data Pipeline 102

Cloudera

JUNE 17, 2022

In this third installment of the Universal Data Distribution blog series, we will take a closer look at how CDF-PC’s new Inbound Connections feature enables universal application connectivity and allows you to build hybrid data pipelines that span the edge, your data center, and one or more public clouds.

Data Pipeline 94

Data Pipeline Building Kafka Java 94

Data Engineering Podcast

OCTOBER 28, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Get started for free at dataengineeringpodcast.com/hightouch. Can you describe what Decodable is and the story behind it?

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? How much end user value does that provide? How much end user value does that provide? Links Datacoral Yahoo!

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

Data Engineering Podcast

MARCH 2, 2020

The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing. Developers can work with the SQL constructs that they are familiar with while automatically getting the durability and reliability that Kafka offers. How is ksqlDB architected?

Kafka 100

Kafka Process PostgreSQL MySQL 100

Data Engineering Podcast

MAY 27, 2018

Your host is Tobias Macey and today I’m interviewing Yair Weinberger about Alooma, a company providing data pipelines as a service Interview Introduction How did you get involved in the area of data management? What is Alooma and what is the origin story? How is the Alooma platform architected?

Data Pipeline 100

Data Pipeline MongoDB Google Cloud Scala 100

Confluent

JULY 23, 2019

The Kafka Summit Program Committee recently published the schedule for the San Francisco event, and there’s quite a bit to look forward to. I remember two to three years back, I spent all my time listening to talks about various ETL architectures in the Pipelines track. Interests evolve over time too. What’s the Time?…and

Kafka 18

Kafka Hadoop Media Software Engineer 18

Confluent

MAY 8, 2019

In anything but the smallest deployment of Apache Kafka ® , there are often going to be multiple clusters of Kafka Connect and KSQL. You don’t want a sudden influx of data from a source upstream to impact other connectors. In this example, there are two clusters of Kafka Connect and two KSQL clusters.

Kafka 89

Kafka Management Hadoop Database 89

Confluent

OCTOBER 9, 2019

Last week, the Kafka Summit hosted nearly 2,000 people from 40 different countries and 595 companies—the largest Summit yet. By the numbers, we got to enjoy four keynote speakers, 56 sessions, 75 speakers, 38 sponsors, and one big party, including the classic Apache Kafka ® ice sculpture, per the traditions handed down to us. (I

Kafka 17

Kafka Software Engineer Software Engineering Data Pipeline 17

Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Christophe Blefari

MARCH 15, 2024

Unlocking Kafka's potential: tackling tail latency with eBPF. Forward thinking Dataviz is hierarchical — Malloy, once again, provides an excellent article about a new way to see data visualisations. Coding data pipelines is faster than renting connector catalogs — This is something I've always believed.

Metadata 272

Metadata Data Software Engineer Datasets 272

Towards Data Science

FEBRUARY 9, 2024

The first phase focuses on building a data pipeline. This involves getting data from an API and storing it in a PostgreSQL database. Overview Let’s break down the data pipeline process step-by-step: Data Streaming: Initially, data is streamed from the API into a Kafka topic.

Kafka 98

Kafka Data Engineering Data Engineer PostgreSQL 98

Confluent

FEBRUARY 28, 2019

Only a little more than one month after the first release, we are happy to announce another milestone for our Kafka integration. Today, you can grab the Kafka Connect Neo4j Sink from Confluent Hub. . Neo4j extension – Kafka sink refresher. Testing the Kafka Connect Neo4j Sink. curl -X POST [link]. jar -f AVRO -e 100000.

Kafka 88

Kafka Java Programming Language Big Data 88

Precisely

MARCH 28, 2025

The data generated was as varied as the departments relying on these applications. Some departments used IBM Db2, while others relied on VSAM files or IMS databases creating complex data governance processes and costly data pipeline maintenance. They chose the Precisely Data Integrity Suites Data Integration Service.

Data Integration 67

Data Integration Pipeline-centric Database-centric Kafka 67

Cloudera

MARCH 14, 2023

In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development. Key requirements for building data pipelines Every data pipeline starts with a business requirement.

Data Pipeline 84

Data Pipeline Designing Kafka Metadata 84

Towards Data Science

MARCH 6, 2023

On-premise and cloud working together to deliver a data product Photo by Toro Tseleng on Unsplash Developing a data pipeline is somewhat similar to playing with lego, you mentalize what needs to be achieved (the data requirements), choose the pieces (software, tools, platforms), and fit them together.

Google Cloud 98

Google Cloud Cloud Storage Data Pipeline Cloud 98

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. Similar to Google in web browsing and Photoshop in image processing, it became a gold standard in data streaming, preferred by 70 percent of Fortune 500 companies. What is Kafka? What Kafka is used for.

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Confluent

OCTOBER 16, 2019

Trains are an excellent source of streaming data—their movements around the network are an unbounded series of events. Using this data, Apache Kafka ® and Confluent Platform can provide the foundations for both event-driven applications as well as an analytical platform. As with any real system, the data has “character.”

Kafka 19

Kafka Building Data Coding 19

Cloudera

SEPTEMBER 26, 2023

Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page. The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog.

Kafka 95

Kafka Technology IT Government 95

Confluent

MARCH 2, 2021

Today, every company is a data company. There are many different data pipeline, integration, and ingestion tools in the market, but before you can feed your data analytics needs, data […].

Kafka 126

Kafka Data Pipeline Data Analytics Data 126

Confluent

JANUARY 20, 2022

A data pipeline is a method for getting data from one system to another, whether for analytics purposes or for storage. Learning the elements that make up this proven architecture […].

Data Pipeline 64

Data Pipeline Kafka Architecture Data 64

Confluent

MAY 15, 2019

Driven by this, we designed and delivered an architecture using Apache Kafka ® and the Confluent Platform. Let’s first discuss the streams of data going into Oracle WMS Cloud: Figure 2. Streaming data into Oracle WMS Cloud. Streaming data out of Oracle WMS Cloud. Streaming data out of Oracle WMS Cloud.

Kafka 92

Kafka ETL Tools Cloud Data Integration 92

Monte Carlo

OCTOBER 31, 2024

AI data engineers are data engineers that are responsible for developing and managing data pipelines that support AI and GenAI data products. Essential Skills for AI Data Engineers Expertise in Data Pipelines and ETL Processes A foundational skill for data engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Confluent

JUNE 11, 2019

It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline. Here in part 4 of the Spring for Apache Kafka Deep Dive blog series, we will cover: Common event streaming topology patterns supported in Spring Cloud Data Flow.

Kafka 86

Kafka Cloud Java MongoDB 86

Workfall

JUNE 27, 2023

It means that there is a high risk of data loss but Apache Kafka solves this because it is distributed and can easily scale horizontally and other servers can take over the workload seamlessly. It offers a unified solution to real-time data needs any organisation might have. This is where Apache Kafka comes in.

Kafka 75

Kafka Architecture AWS Transportation 75

Towards Data Science

JANUARY 7, 2024

Streamline Data Pipelines: How to Use WhyLogs with PySpark for Effective Data Profiling and Validation Photo by Evan Dennis on Unsplash Data pipelines, made by data engineers or machine learning engineers, do more than just prepare data for reports or training models.

Data Pipeline 98

Data Pipeline Hospitality Data Validation Datasets 98

Data Engineering Weekly

FEBRUARY 23, 2025

link] Yelp: Revenue Automation Series: Building Revenue Data Pipeline Yelp writes about its journey to automate revenue recognition by building a revenue data pipeline. link] Apache Kafka: KIP-932 - Queues for Kafka One exciting weekend read for me was the KIP-932 proposal to add queue guarantees to Apache Kafka.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Confluent

AUGUST 11, 2020

Building a Cloud ETL Pipeline on Confluent Cloud shows you how to build and deploy a data pipeline entirely in the cloud. However, not all databases can be in the […].

Database 103

Database Kafka Data Pipeline Cloud 103

Data Engineering Weekly

APRIL 20, 2025

With the chatGPTs as your knowledge assistance, I hope to get this time around :-) [link] TopicPartition: KIP-1150 in Apache Kafka is a big deal (Diskless Topics) When I saw the KIP-1150 proposal, I was like, okay, finally it is happening. Kafka is probably the most reliable data infrastructure in the modern data era.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Data Engineering Podcast

FEBRUARY 4, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Starburst Logo]([link] This episode is brought to you by Starburst - a data lake analytics platform for data engineers who are battling to build and scale high quality data pipelines on the data lake.

SQL 173

SQL Data Lake High Quality Data Machine Learning 173