Data Ingestion, Java and MongoDB - Data Engineering Digest

Comparing Snowflake Data Ingestion Methods with Striim

Striim

NOVEMBER 13, 2023

Introduction In the fast-evolving world of data integration, Striim’s collaboration with Snowflake stands as a beacon of innovation and efficiency. Striim’s integration with Snowpipe Streaming represents a significant advancement in real-time data ingestion into Snowflake.

Data Ingestion

Data Ingestion Utilities Data Integration Data

Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet

Data Engineering Podcast

NOVEMBER 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake

Data Lake Data Ingestion MongoDB Google Cloud

An Exploration Of The Expectations, Ecosystem, and Realities Of Real-Time Data Applications

Data Engineering Podcast

AUGUST 21, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Lambda Architecture

Lambda Architecture MongoDB Scala MySQL

Webinars

Prepare Now: 2025s Must-Know Trends For Product And Data Leaders

MORE WEBINARS

Updates, Inserts, Deletes: Comparing Elasticsearch and Rockset for Real-Time Data Ingest

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. That’s because Elasticsearch can only write data to one index.

Data Ingestion

Data Ingestion Kafka Relational Database PostgreSQL

Using Elasticsearch to Offload Real-Time Analytics from MongoDB

Rockset

NOVEMBER 12, 2020

Offloading analytics from MongoDB establishes clear isolation between write-intensive and read-intensive operations. In most scenarios, MongoDB can be used as the primary data storage for write-only operations and as support for quick data ingestion. Monstache is also available as a sync daemon and a container.

MongoDB

MongoDB NoSQL Data Pipeline Data Storage

Going From Transactional To Analytical And Self-managed To Cloud On One Database With MariaDB

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database

Database MySQL Cloud MongoDB

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Data Engineering Podcast

NOVEMBER 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB

MongoDB Scala MySQL Machine Learning

Discover And De-Clutter Your Unstructured Data With Aparavi

Data Engineering Podcast

JUNE 12, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Unstructured Data

Unstructured Data MongoDB Scala MySQL

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Data Engineering Podcast

SEPTEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Pipeline

Data Pipeline Building MongoDB Scala

Level Up Your Data Platform With Active Metadata

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata

Metadata MongoDB Scala MySQL

Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast

Data Engineering Podcast

JULY 17, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineer

Data Engineer Data Engineering Engineering MongoDB

Collecting And Retaining Contextual Metadata For Powerful And Effective Data Discovery

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata

Metadata MongoDB Scala MySQL

Taking A Look Under The Hood At CreditKarma's Data Platform

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB

MongoDB Google Cloud Scala MySQL

Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Engineering

Engineering MongoDB Scala MySQL

An Exploration Of The Open Data Lakehouse And Dremio's Contribution To The Ecosystem

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake

Data Lake Food MongoDB Scala

Power Your Real-Time Analytics Without The Headache Using Fivetran's Change Data Capture Integrations

Data Engineering Podcast

SEPTEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Food

Food MongoDB Scala MySQL

Operational Analytics To Increase Efficiency For Multi-Location Businesses With OpsAnalitica

Data Engineering Podcast

SEPTEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Hospitality

Hospitality Food MongoDB Scala

Alumni Of AirBnB's Early Years Reflect On What They Learned About Building Data Driven Organizations

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building

Building MongoDB Scala MySQL

Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Analysis

Data Analysis MongoDB Algorithm Scala

Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster

Data Engineering Podcast

JULY 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB

MongoDB Scala MySQL Data Lake

Maintain Your Data Engineers' Sanity By Embracing Automation

Data Engineering Podcast

JULY 10, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineer

Data Engineer Data Engineering Engineering MongoDB

Be Confident In Your Data Integration By Quickly Validating Matching Records With data-

Data Engineering Podcast

JULY 3, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Integration

Data Integration MongoDB Scala MySQL

Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault

Data Engineering Podcast

JUNE 5, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Security

Data Security Metadata MongoDB Scala

Investing In Understanding The Customer Journey At American Express

Data Engineering Podcast

OCTOBER 9, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Food

Food MongoDB Scala MySQL

Make Data Lineage A Ubiquitous Part Of Your Work By Simplifying Its Implementation With Alvin

Data Engineering Podcast

OCTOBER 2, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

IT

IT Food PostgreSQL MongoDB

Bringing Automation To Data Labeling For Machine Learning With Watchful

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning

Machine Learning Pipeline-centric Database-centric MongoDB

Optimize Your Machine Learning Development And Serving With The Open Source Vector Database Milvus

Data Engineering Podcast

AUGUST 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning

Machine Learning Database MySQL PostgreSQL

Strategies And Tactics For A Successful Master Data Management Implementation

Data Engineering Podcast

JUNE 26, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Management

Data Management Management MongoDB Scala

Introduce Climate Analytics Into Your Data Platform Without The Heavy Lifting Using Sust Global

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB

MongoDB Scala MySQL Machine Learning

15+ Best Data Engineering Tools to Explore in 2023

Knowledge Hut

APRIL 25, 2023

Here are some essential skills for data engineers when working with data engineering tools. Strong programming skills: Data engineers should have a good grasp of programming languages like Python, Java, or Scala, which are commonly used in data engineering.

Data Engineer

Data Engineer Data Engineering Engineering Google Cloud

The Good and the Bad of Hadoop Big Data Framework

AltexSoft

JULY 29, 2022

Apache Hadoop is an open-source Java-based framework that relies on parallel processing and distributed storage for analyzing massive datasets. Developed in 2006 by Doug Cutting and Mike Cafarella to run the web crawler Apache Nutch, it has become a standard for Big Data analytics. MongoDB: an NoSQL database with additional features.

Hadoop

Hadoop Big Data Google Cloud NoSQL

SQL and Complex Queries Are Needed for Real-Time Analytics

Rockset

MAY 17, 2022

Their query languages, whether SQL-like variants such as CQL (Cassandra) and Druid SQL or wholly custom languages such as MQL (MongoDB), poorly support joins and other complex query commands that are standard to SQL , if they support them at all. Instant scaleup of data writes or queries to handle bursts of data.

SQL

SQL NoSQL Hadoop MongoDB

Sqoop vs. Flume Battle of the Hadoop ETL tools

ProjectPro

OCTOBER 28, 2015

Apache Hadoop is synonymous with big data for its cost-effectiveness and its attribute of scalability for processing petabytes of data. Data analysis using hadoop is just half the battle won. Getting data into the Hadoop cluster plays a critical role in any big data deployment. Sqoop makes data analysis efficient.

ETL Tools

ETL Tools Hadoop Relational Database Unstructured Data

20+ Data Engineering Projects for Beginners with Source Code

ProjectPro

AUGUST 24, 2021

Data Engineering Project for Beginners If you are a newbie in data engineering and are interested in exploring real-world data engineering projects, check out the list of data engineering project examples below. This big data project discusses IoT architecture with a sample use case.

Data Engineer

Data Engineer Data Engineering Coding Project

100+ Big Data Interview Questions and Answers 2023

ProjectPro

JANUARY 31, 2023

There are three steps involved in the deployment of a big data model: Data Ingestion: This is the first step in deploying a big data model - Data ingestion, i.e., extracting data from multiple data sources. Query Surge helps us automate our manual efforts in Big Data testing.

Big Data

Big Data Hadoop Relational Database AWS

Data Engineer Learning Path, Career Track & Roadmap for 2023

ProjectPro

JANUARY 19, 2022

Data Engineering Requirements Here is a list of skills needed to become a data engineer: Highly skilled at graduation-level mathematics. Good skills in computer programming languages like R, Python, Java, C++, etc. High efficiency in advanced probability and statistics.

Data Engineer

Data Engineer Data Engineering Engineering Amazon Web Services

Comparing Snowflake Data Ingestion Methods with Striim

Tame The Entropy In Your Data Stack And Prevent Failures With Sifflet

Webinars

Trending Sources

An Exploration Of The Expectations, Ecosystem, and Realities Of Real-Time Data Applications

Webinars

Updates, Inserts, Deletes: Comparing Elasticsearch and Rockset for Real-Time Data Ingest

Using Elasticsearch to Offload Real-Time Analytics from MongoDB

Going From Transactional To Analytical And Self-managed To Cloud On One Database With MariaDB

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Discover And De-Clutter Your Unstructured Data With Aparavi

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Level Up Your Data Platform With Active Metadata

Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast

Collecting And Retaining Contextual Metadata For Powerful And Effective Data Discovery

Taking A Look Under The Hood At CreditKarma's Data Platform

Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt

An Exploration Of The Open Data Lakehouse And Dremio's Contribution To The Ecosystem

Power Your Real-Time Analytics Without The Headache Using Fivetran's Change Data Capture Integrations

Operational Analytics To Increase Efficiency For Multi-Location Businesses With OpsAnalitica

Alumni Of AirBnB's Early Years Reflect On What They Learned About Building Data Driven Organizations

Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda

Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster

Maintain Your Data Engineers' Sanity By Embracing Automation

Be Confident In Your Data Integration By Quickly Validating Matching Records With data-

Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault

Investing In Understanding The Customer Journey At American Express

Make Data Lineage A Ubiquitous Part Of Your Work By Simplifying Its Implementation With Alvin

Bringing Automation To Data Labeling For Machine Learning With Watchful

Optimize Your Machine Learning Development And Serving With The Open Source Vector Database Milvus

Strategies And Tactics For A Successful Master Data Management Implementation

Introduce Climate Analytics Into Your Data Platform Without The Heavy Lifting Using Sust Global

15+ Best Data Engineering Tools to Explore in 2023

The Good and the Bad of Hadoop Big Data Framework

SQL and Complex Queries Are Needed for Real-Time Analytics

Sqoop vs. Flume Battle of the Hadoop ETL tools

20+ Data Engineering Projects for Beginners with Source Code

100+ Big Data Interview Questions and Answers 2023

Data Engineer Learning Path, Career Track & Roadmap for 2023

Stay Connected