Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 130

Metadata MongoDB MySQL Scala 130

Striim

NOVEMBER 13, 2023

Introduction In the fast-evolving world of data integration, Striim’s collaboration with Snowflake stands as a beacon of innovation and efficiency. Snowpipe Streaming: Unleashing Real-Time Data Integration and AI Snowpipe Streaming, when teamed up with Striim, is kind of like a superhero for real-time data needs.

Data Ingestion 59

Data Ingestion Data Integration Utilities Data 59

ProjectPro

JUNE 6, 2025

AWS (Amazon Web Services) is the leading global cloud platform, offering over 200 fully featured services from data centers worldwide. With over 1 million active enterprise customers and a thriving ecosystem of partners and third-party software products, AWS is at the forefront of cloud computing.

AWS 60

AWS Project Food Cloud Computing 60

Data Engineering Podcast

NOVEMBER 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 130

MongoDB MySQL Scala Data Lake 130

Data Engineering Podcast

AUGUST 21, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

ProjectPro

JUNE 6, 2025

Experts predict that by 2025, the global big data and data engineering market will reach $125.89 billion, and those with skills in cloud-based ETL tools and distributed systems will be in the highest demand. As more organizations shift to the cloud, the demand for ETL engineers with expertise in these platforms is soaring.

Data Engineer 40

Data Engineer Data Engineering Engineering ETL Tools 40

Data Engineering Podcast

JULY 24, 2022

In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0 release, and the new features coming with Dagster Cloud’s general availability. Data teams are increasingly under pressure to deliver. and cloud to GA?

MongoDB 100

MongoDB MySQL Scala Data Lake 100

Data Engineering Podcast

JULY 17, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5% Why migrate?

MongoDB 100

MongoDB MySQL Scala Google Cloud 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Food 100

Food MongoDB MySQL Scala 100

Data Engineering Podcast

JULY 3, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Integration 100

Data Integration MongoDB MySQL Scala 100

Data Engineering Podcast

JULY 10, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Analysis 100

Data Analysis MongoDB MySQL Scala 100

Data Engineering Podcast

SEPTEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Hospitality 100

Hospitality Food MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Lake 100

Data Lake Food MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. In fact, while only 3.5%

Food 100

Food MongoDB MySQL Scala 100

Data Engineering Podcast

AUGUST 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Machine Learning 100

Machine Learning Database MySQL MongoDB 100

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 100

Metadata MongoDB MySQL Scala 100

Data Engineering Podcast

JUNE 26, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Management 100

Data Management Management MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 2, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

IT 100

IT Food MongoDB PostgreSQL 100

Data Engineering Podcast

JUNE 5, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Security 100

Data Security Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Engineering 100

Engineering MongoDB MySQL Scala 100

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. Elasticsearch was designed for log analytics where data is not frequently changing, posing additional challenges when dealing with transactional data.

Data Ingestion 52

Data Ingestion Kafka PostgreSQL Relational Database 52

Rockset

NOVEMBER 12, 2020

Offloading analytics from MongoDB establishes clear isolation between write-intensive and read-intensive operations. In most scenarios, MongoDB can be used as the primary data storage for write-only operations and as support for quick data ingestion. Monstache is also available as a sync daemon and a container.

MongoDB 40

MongoDB NoSQL Data Pipeline Data Storage 40

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

ProjectPro

JUNE 6, 2025

Consequently, data engineers implement checkpoints so that no event is missed or processed twice. It not only consumes more memory but also slackens data transfer. Modern cloud-based data pipelines are agile and elastic to automatically scale compute and storage resources. ADF does not store any data on its own.

Data Pipeline 40

Data Pipeline Architecture Kafka Data Lake 40

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 100

MongoDB MySQL Scala Data Ingestion 100

Rockset

NOVEMBER 23, 2021

We built Rockset with the mission to make real-time analytics easy and affordable in the cloud. We put our users first and obsess about helping our users achieve speed, scale and simplicity in their modern real-time data stack (some of which I discuss in depth below). Change data capture streams. The problem?

MongoDB 59

MongoDB Data Ingestion PostgreSQL SQL 59

ProjectPro

JUNE 6, 2025

Traditional data tools cannot handle this massive volume of complex data, so several unique Big Data software tools and architectural solutions have been developed to handle this task. Big Data Tools extract and process data from multiple data sources. Why Are Big Data Tools Valuable to Data Professionals?

Big Data Tools 40

Big Data Tools Big Data Hadoop Kafka 40

Rockset

JANUARY 9, 2023

Modern Snack-Sized Sales Training At ConveYour , we provide automated sales training via the cloud. Technical Challenges Our original data infrastructure was built around an on-premises MongoDB database that ingested and stored all user transaction data. First is its speed at data ingestion.

MySQL 52

MySQL MongoDB Recruitment Data Ingestion 52

ProjectPro

JUNE 6, 2025

Types of AWS Databases AWS provides various database services, such as Relational Databases Non-Relational or NoSQL Databases Other Cloud Databases ( In-memory and Graph Databases). Relational Databases Relational databases form the backbone of modern data storage and management systems, powering various applications across industries.

AWS 40

AWS Database Amazon Web Services MySQL 40

ProjectPro

JUNE 6, 2025

There are three steps involved in the deployment of a big data model: Data Ingestion: This is the first step in deploying a big data model - Data ingestion, i.e., extracting data from multiple data sources. When to use MapReduce with Big Data. For example – MongoDB.

Big Data 40

Big Data Hadoop Relational Database NoSQL 40

DareData

JANUARY 30, 2023

Our goal is to help data scientists better manage their models deployments or work more effectively with their data engineering counterparts, ensuring their models are deployed and maintained in a robust and reliable way. AWS Glue: A fully managed data orchestrator service offered by Amazon Web Services (AWS).

Data Engineer 52

Data Engineer Data Engineering Engineering NoSQL 52

ProjectPro

JUNE 6, 2025

Data Collection & Preprocessing Gather historical sales data, product demand reports, and macroeconomic indicators. Clean and preprocess raw data, handle missing values and seasonality trends. Data Collection & Preprocessing Aggregate historical sales, suppliers, and warehouse raw data.

Data Science 40

Data Science Project Management Transportation 40

Striim

OCTOBER 11, 2024

As you’ll see by taking a look at this data pipeline example, the complexity and design of a pipeline varies depending on intended use. For instance, Macy’s streams change data from on-premises databases to Google Cloud. Another excellent data pipeline example is American Airlines’ work with Striim.

Data Pipeline 52

Data Pipeline MongoDB Unstructured Data Data Lake 52

AltexSoft

MAY 12, 2023

A loose schema allows for some data structure flexibility while maintaining a general organization. Semi-structured data is typically stored in NoSQL databases, such as MongoDB, Cassandra, and Couchbase, following hierarchical or graph data models. MongoDB, Cassandra), and big data processing frameworks (e.g.,

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

AltexSoft

MAY 14, 2021

Big Data analytics encompasses the processes of collecting, processing, filtering/cleansing, and analyzing extensive datasets so that organizations can use them to develop, grow, and produce better products. Big Data analytics processes and tools. Data ingestion. Data storage and processing.

Big Data 78

Big Data Data Analytics IT NoSQL 78