Data Pipeline and MongoDB - Data Engineering Digest

Kafka to MongoDB: Building a Streamlined Data Pipeline

Analytics Vidhya

FEBRUARY 28, 2024

Handling and processing the streaming data is the hardest work for Data Analysis. We know that streaming data is data that is emitted at high volume […] The post Kafka to MongoDB: Building a Streamlined Data Pipeline appeared first on Analytics Vidhya.

MongoDB

MongoDB Data Pipeline Kafka Building

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. Data stacks are becoming more and more complex.

Data Pipeline

Data Pipeline Building MongoDB MySQL

The Alooma Data Pipeline With CTO Yair Weinberger - Episode 33

Data Engineering Podcast

MAY 27, 2018

Your host is Tobias Macey and today I’m interviewing Yair Weinberger about Alooma, a company providing data pipelines as a service Interview Introduction How did you get involved in the area of data management? What is Alooma and what is the origin story? How is the Alooma platform architected?

Data Pipeline

Data Pipeline MongoDB Google Cloud Scala

Webinars

Agent Tooling: Connecting AI to Your Tools, Systems & Data

How to Modernize Manufacturing Without Losing Control

Mastering Apache Airflow® 3.0: What’s New (and What’s Next) for Data Orchestration

MORE WEBINARS

What is an AI Data Engineer? 4 Important Skills, Responsibilities, & Tools

Monte Carlo

OCTOBER 31, 2024

AI data engineers are data engineers that are responsible for developing and managing data pipelines that support AI and GenAI data products. Essential Skills for AI Data Engineers Expertise in Data Pipelines and ETL Processes A foundational skill for data engineers?

Data Engineer

Data Engineer Data Engineering Engineering Unstructured Data

What is a Data Pipeline (and 7 Must-Have Features of Modern Data Pipelines)

Striim

OCTOBER 11, 2024

A well-executed data pipeline can make or break your company’s ability to leverage real-time insights and stay competitive. Thriving in today’s world requires building modern data pipelines that make moving data and extracting valuable insights quick and simple. What is a Data Pipeline?

Data Pipeline

Data Pipeline MongoDB Unstructured Data Data Lake

Top 10 MongoDB Career Options in 2024 [Job Opportunities]

Knowledge Hut

MARCH 22, 2024

I am here to discuss MongoDB job opportunities for you in 2024 and the wide spectrum of options that it provides. But first, let’s discuss MongoDB a bit. MongoDB is the fourth most popular Database Management System (DBMS). Significantly, MongoDB has witnessed an influencing growth of 163% in the last two years!

MongoDB

MongoDB Amazon Web Services Computer Science Education

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Links RavenDB RSS Object Relational Mapper (ORM) Relational Database NoSQL CouchDB Navigational Database MongoDB Redis Neo4J Cassandra Column-Family SQLite LevelDB Firebird DB fsync Esent DB?

Non-relational Database

Non-relational Database Relational Database Database Designing

Five Ways to Run Analytics on MongoDB – Their Pros and Cons

Rockset

FEBRUARY 2, 2022

MongoDB is a top database choice for application development. Developers choose this database because of its flexible data model and its inherent scalability as a NoSQL database. MongoDB wasn’t originally developed with an eye on high performance for analytics. Yet, analytics is now a vital part of modern data applications.

MongoDB

MongoDB NoSQL Data Warehouse BI

Astronomer with Ry Walker - Episode 6

Data Engineering Podcast

AUGUST 6, 2017

Summary Building a data pipeline that is reliable and flexible is a difficult task, especially when you have a small team. Astronomer is a platform that lets you skip straight to processing your valuable business data. Interview Introduction How did you first get involved in the area of data management?

PostgreSQL

PostgreSQL MongoDB Data Pipeline Kafka

Build Data Products Without A Data Team Using AgileData

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building

Building Metadata MongoDB MySQL

Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence

Business Intelligence Metadata MongoDB MySQL

Making Sense Of The Technical And Organizational Considerations Of Data Contracts

Data Engineering Podcast

DECEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode.

Metadata

Metadata Business Intelligence Data Lake BI

A Look At The Data Systems Behind The Gameplay For League Of Legends

Data Engineering Podcast

NOVEMBER 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems

Systems Metadata Data Pipeline MongoDB

Increase Your Odds Of Success For Analytics And AI Through More Effective Knowledge Management With AlignAI

Data Engineering Podcast

DECEMBER 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode.

Management

Management Metadata Business Intelligence Data Lake

Using Product Driven Development To Improve The Productivity And Effectiveness Of Your Data Teams

Data Engineering Podcast

DECEMBER 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode.

Data Lake

Data Lake Data Warehouse Data Pipeline MongoDB

Connect Data From Legacy Databases Anywhere to MongoDB in Real Time With Confluent and Apache Kafka®

Confluent

DECEMBER 31, 1969

Connect any database to MongoDB using Confluent's cloud-native data streaming platform. Modernize any database, build streaming data pipelines, and empower real-time data in minutes.

MongoDB

MongoDB Database Kafka Data Pipeline

Build Better Data Products By Creating Data, Not Consuming It

Data Engineering Podcast

NOVEMBER 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building

Building IT Metadata MongoDB

Simple And Scalable Encryption Of Data In Use For Analytics And Machine Learning With Opaque Systems

Data Engineering Podcast

DECEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. Rudderstack : ![Rudderstack]([link]

Machine Learning

Machine Learning Systems Data Lake Data Warehouse

An Exploration Of The Expectations, Ecosystem, and Realities Of Real-Time Data Applications

Data Engineering Podcast

AUGUST 21, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Lambda Architecture

Lambda Architecture MongoDB MySQL Scala

Making The Total Cost Of Ownership For External Data Manageable With Crux

Data Engineering Podcast

JULY 17, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Tired of deploying bad data?

Data Management

Data Management Management Metadata MongoDB

Using Elasticsearch to Offload Real-Time Analytics from MongoDB

Rockset

NOVEMBER 12, 2020

Offloading analytics from MongoDB establishes clear isolation between write-intensive and read-intensive operations. In most scenarios, MongoDB can be used as the primary data storage for write-only operations and as support for quick data ingestion. If you have static data in MongoDB, you may need a one-time sync.

MongoDB

MongoDB NoSQL Data Pipeline Data Storage

Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata

Metadata Unstructured Data MongoDB MySQL

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Data Engineering Podcast

NOVEMBER 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB

MongoDB MySQL Scala Machine Learning

Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet

Data Engineering Podcast

NOVEMBER 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake

Data Lake Data Ingestion MongoDB MySQL

How To Bring Agile Practices To Your Data Projects

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Project

Project Metadata MongoDB MySQL

Database Refactoring Patterns with Pramod Sadalage - Episode 22

Data Engineering Podcast

MARCH 11, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.

Database

Database MongoDB NoSQL Database Design

Offload Real-Time Reporting and Analytics from MongoDB Using PostgreSQL

Rockset

SEPTEMBER 3, 2020

MongoDB’s Advantages & Disadvantages MongoDB has comprehensive aggregation capabilities. You can run many analytic queries on MongoDB without exporting your data to a third-party tool. In this situation, the MongoDB cluster doesn’t have to keep up with the read requests. This is never a good thing.

MongoDB

MongoDB PostgreSQL SQL Database

An Exploration Of What Data Automation Can Provide To Data Engineers And Ascend's Journey To Make It A Reality

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineer

Data Engineer Data Engineering MongoDB Metadata

Adopting Real-Time Data At Organizations Of Every Size

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake

Data Lake MongoDB MySQL Data Warehouse

Level Up Your Data Platform With Active Metadata

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata

Metadata MongoDB MySQL Scala

Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform

Data Engineering Podcast

JUNE 26, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Datasets

Datasets Unstructured Data Metadata MongoDB

Data Pipeline- Definition, Architecture, Examples, and Use Cases

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline

Data Pipeline Architecture Kafka AWS

Business Intelligence In The Palm Of Your Hand With Zing Data

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Struggling with broken pipelines?

Business Intelligence

Business Intelligence Metadata BI MongoDB

Run Your Applications Worldwide Without Worrying About The Database With Planetscale

Data Engineering Podcast

DECEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database

Database MySQL Data Lake MongoDB

Analyze Massive Data At Interactive Speeds With The Power Of Bitmaps Using FeatureBase

Data Engineering Podcast

NOVEMBER 27, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake

Data Lake Data Warehouse MongoDB Data Pipeline

TimescaleDB: Fast And Scalable Timeseries with Ajay Kulkarni and Mike Freedman - Episode 18

Data Engineering Podcast

FEBRUARY 11, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.

PostgreSQL

PostgreSQL NoSQL Google Cloud MongoDB

Leave Your Data Where It Is And Automate Feature Extraction With Molecula

Data Engineering Podcast

MARCH 8, 2021

Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.

IT

IT Data Warehouse MongoDB Kafka

Charting the Path of Riskified's Data Platform Journey

Data Engineering Podcast

JULY 10, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Tired of deploying bad data?

Metadata

Metadata MongoDB MySQL Machine Learning

Building A Shared Understanding Of Data Assets In A Business Through A Single Pane Of Glass With Workstream

Data Engineering Podcast

SEPTEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building

Building Metadata MongoDB MySQL

Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language

Data Engineering Podcast

SEPTEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building

Building Metadata MongoDB MySQL

Gain Visibility And Insight Into Your Supply Chains Through Operational Analytics Powered By Roambee

Data Engineering Podcast

OCTOBER 2, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata

Metadata Electronics MongoDB MySQL

Speeding Up The Time To Insight For Supply Chains And Logistics With The Pathway Database That Thinks

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database

Database Metadata MongoDB MySQL

Supporting And Expanding The Arrow Ecosystem For Fast And Efficient Data Processing At Voltron Data

Data Engineering Podcast

NOVEMBER 27, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Process

Data Process Process Metadata Business Intelligence

Adding Support For Distributed Transactions To The Redpanda Streaming Engine

Data Engineering Podcast

OCTOBER 5, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines.

Engineering

Engineering MongoDB Kafka Data Lake

Kafka to MongoDB: Building a Streamlined Data Pipeline

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Webinars

Trending Sources

The Alooma Data Pipeline With CTO Yair Weinberger - Episode 33

Webinars

What is an AI Data Engineer? 4 Important Skills, Responsibilities, & Tools

What is a Data Pipeline (and 7 Must-Have Features of Modern Data Pipelines)

Top 10 MongoDB Career Options in 2024 [Job Opportunities]

Designing A Non-Relational Database Engine

Five Ways to Run Analytics on MongoDB – Their Pros and Cons

Astronomer with Ry Walker - Episode 6

Build Data Products Without A Data Team Using AgileData

Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense

Making Sense Of The Technical And Organizational Considerations Of Data Contracts

A Look At The Data Systems Behind The Gameplay For League Of Legends

Increase Your Odds Of Success For Analytics And AI Through More Effective Knowledge Management With AlignAI

Using Product Driven Development To Improve The Productivity And Effectiveness Of Your Data Teams

Connect Data From Legacy Databases Anywhere to MongoDB in Real Time With Confluent and Apache Kafka®

Build Better Data Products By Creating Data, Not Consuming It

Simple And Scalable Encryption Of Data In Use For Analytics And Machine Learning With Opaque Systems

An Exploration Of The Expectations, Ecosystem, and Realities Of Real-Time Data Applications

Making The Total Cost Of Ownership For External Data Manageable With Crux

Using Elasticsearch to Offload Real-Time Analytics from MongoDB

Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet

How To Bring Agile Practices To Your Data Projects

Database Refactoring Patterns with Pramod Sadalage - Episode 22

Offload Real-Time Reporting and Analytics from MongoDB Using PostgreSQL

An Exploration Of What Data Automation Can Provide To Data Engineers And Ascend's Journey To Make It A Reality

Adopting Real-Time Data At Organizations Of Every Size

Level Up Your Data Platform With Active Metadata

Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform

Data Pipeline- Definition, Architecture, Examples, and Use Cases

Business Intelligence In The Palm Of Your Hand With Zing Data

Run Your Applications Worldwide Without Worrying About The Database With Planetscale

Analyze Massive Data At Interactive Speeds With The Power Of Bitmaps Using FeatureBase

TimescaleDB: Fast And Scalable Timeseries with Ajay Kulkarni and Mike Freedman - Episode 18

Leave Your Data Where It Is And Automate Feature Extraction With Molecula

Charting the Path of Riskified's Data Platform Journey

Building A Shared Understanding Of Data Assets In A Business Through A Single Pane Of Glass With Workstream

Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language

Gain Visibility And Insight Into Your Supply Chains Through Operational Analytics Powered By Roambee

Speeding Up The Time To Insight For Supply Chains And Logistics With The Pathway Database That Thinks

Supporting And Expanding The Arrow Ecosystem For Fast And Efficient Data Processing At Voltron Data

Adding Support For Distributed Transactions To The Redpanda Streaming Engine

Stay Connected