Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 130

Metadata MongoDB MySQL Scala 130

Data Engineering Podcast

AUGUST 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Cloudera

OCTOBER 11, 2021

Modak’s Nabu is a born in the cloud, cloud-neutral integrated data engineering platform designed to accelerate the journey of enterprises to the cloud. The platform converges data cataloging, data ingestion, data profiling, data tagging, data discovery, and data exploration into a unified platform, driven by metadata.

Data Engineer 95

Data Engineer Data Engineering Cloud Engineering 95

Data Engineering Podcast

SEPTEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Pipeline 130

Data Pipeline Building MongoDB MySQL 130

Data Engineering Podcast

AUGUST 21, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Data Engineering Podcast

JULY 17, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 100

Metadata MongoDB MySQL Scala 100

Data Engineering Podcast

MAY 29, 2022

report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. In fact, while only 3.5% That’s where our friends at Ascend.io

Database 100

Database Architecture Data Architecture PostgreSQL 100

Data Engineering Podcast

JUNE 5, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Security 100

Data Security Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 3, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Integration 100

Data Integration MongoDB MySQL Scala 100

Data Engineering Podcast

JULY 10, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Data Engineering Podcast

JULY 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 100

MongoDB MySQL Scala Data Lake 100

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Analysis 100

Data Analysis MongoDB Algorithm MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Hospitality 100

Hospitality Food MongoDB MySQL 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Food 100

Food MongoDB MySQL Scala 100

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Lake 100

Data Lake Food MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Engineering 100

Engineering MongoDB MySQL Scala 100

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 100

MongoDB MySQL Google Cloud Scala 100

Data Engineering Podcast

JUNE 26, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Management 100

Data Management Management MongoDB MySQL 100

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Data Engineering Podcast

OCTOBER 2, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

OCTOBER 9, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Food 100

Food MongoDB MySQL Scala 100

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 100

MongoDB MySQL Scala Machine Learning 100

Cloudera

SEPTEMBER 28, 2020

This customer’s workloads leverage batch processing of data from 100+ backend database sources like Oracle, SQL Server, and traditional Mainframes using Syncsort. Data Science and machine learning workloads using CDSW. The customer is a heavy user of Kafka for data ingestion. Postgres 10, MySQL 5.7 or Ubuntu 18.04.

Cloud 132

Cloud Kafka Professional Services Metadata 132

Rockset

JANUARY 9, 2023

Technical Challenges Our original data infrastructure was built around an on-premises MongoDB database that ingested and stored all user transaction data. However, the biggest reason was simply that MySQL is not designed for high-speed analytics. First is its speed at data ingestion. It took just one week.

MySQL 52

MySQL MongoDB Recruitment Data Ingestion 52

Booking.com Engineering

DECEMBER 2, 2022

From data ingestion, data science, to our ad bidding[2], GCP is an accelerant in our development cycle, sometimes reducing time-to-market from months to weeks. Data Ingestion and Analytics at Scale Ingestion of performance data, whether generated by a search provider or internally, is a key input for our algorithms.

Systems 52

Systems Cloud MySQL Relational Database 52

DareData

JANUARY 30, 2023

Our goal is to help data scientists better manage their models deployments or work more effectively with their data engineering counterparts, ensuring their models are deployed and maintained in a robust and reliable way. DigDag: An open-source orchestrator for data engineering workflows.

Data Engineer 52

Data Engineer Data Engineering NoSQL Engineering 52

ProjectPro

OCTOBER 28, 2015

Apache Hadoop is synonymous with big data for its cost-effectiveness and its attribute of scalability for processing petabytes of data. Data analysis using hadoop is just half the battle won. Getting data into the Hadoop cluster plays a critical role in any big data deployment. then you are on the right page.

ETL Tools 52

ETL Tools Hadoop Relational Database Unstructured Data 52

Rockset

NOVEMBER 23, 2021

The other area where Rockset shines is that it is built to handle both time-series data streams as well as as CDC streams with updates, inserts and deletes, making it possible to stay in real-time sync with databases like DynamoDB, MongoDB, PostgreSQL, MySQL without any reindexing overhead.

MongoDB 59

MongoDB Data Ingestion SQL PostgreSQL 59

ProjectPro

FEBRUARY 8, 2023

This serverless data integration service can automatically and quickly discover structured or unstructured enterprise data when stored in data lakes in Amazon S3, data warehouses in Amazon Redshift, and other databases that are a component of the Amazon Relational Database Service.

AWS 98

AWS Scala Metadata Data Lake 98

Striim

AUGUST 7, 2024

Implementation Details Here is the high level architecture and data flow of the solution: Generate POS Data The POS (Point of Sales) Training dataset was synthetically created using a Python script and then loaded into MySQL. POS transactions training data span 79 days starting from (2024-02-01 to 2024-04-20).

IT 52

IT Entertainment MySQL Raw Data 52

Rockset

AUGUST 30, 2021

A Quick Primer on Indexing in Rockset Rockset allows users to connect real-time data sources — data streams (Kafka, Kinesis), OLTP databases (DynamoDB, MongoDB, MySQL, PostgreSQL) and also data lakes (S3, GCS) — using built-in connectors. In example above, these base aggregate metrics are count(*) and sum(error_flag).

SQL 52

SQL Kafka MongoDB MySQL 52

Rockset

OCTOBER 4, 2022

Flink, Kafka and MySQL. As real-time analytics databases, Rockset and ClickHouse are built for low-latency analytics on large data sets. They possess distributed architectures that allow for scalability to handle performance or data volume requirements.

MySQL 52

MySQL Kafka Aggregated Data Architecture 52

ProjectPro

AUGUST 24, 2021

Data Engineering Project for Beginners If you are a newbie in data engineering and are interested in exploring real-world data engineering projects, check out the list of data engineering project examples below. This big data project discusses IoT architecture with a sample use case.

Data Engineer 52

Data Engineer Data Engineering Coding Project 52

Cloudera

SEPTEMBER 17, 2018

Here are some highlights: Data Ingest. Most data is ingested through data engineering pipelines. But for an SQL user, it is also common to have “data laying around” – some flat files on S3, some tables in an external DB. Data Discovery and Exploration.

Data Warehouse 53

Data Warehouse SQL Portfolio MySQL 53

Knowledge Hut

MARCH 19, 2024

Subsystems popular databases like MySQL, PostgreSQL, and Microsoft SQL Server, eliminating manual database management tasks like hardware provisioning, patching, and backups. It makes real-time streaming data collection, processing, and analytics possible for timely insight and decision-making for businesses.

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Silectis

JANUARY 3, 2021

Data Engineering Data engineering is a process by which data engineers make data useful. Data engineers design, build, and maintain data pipelines that transform data from a raw state to a useful one, ready for analysis or data science modeling.

Data Engineer 40

Data Engineer Data Engineering Engineering Hadoop 40