Data Engineering Weekly

MARCH 5, 2025

While the Iceberg itself simplifies some aspects of data management, the surrounding ecosystem introduces new challenges: Small File Problem (Revisited): Like Hadoop, Iceberg can suffer from small file problems. Data ingestion tools often create numerous small files, which can degrade performance during query execution.

Hadoop 57

Hadoop Metadata Data Ingestion Data Governance 57

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. In fact, while only 3.5%

Data Lake 130

Data Lake Data Ingestion MongoDB MySQL 130

Snowflake

APRIL 9, 2025

Sherloq Data management is critical when building internal gen AI applications, but it remains a challenge for most companies: Creating a verified source of truth and keeping it up to date with the latest documentation is a highly manual, high-effort task.

Pharmaceutical 97

Pharmaceutical Manufacturing Data Ingestion SQL 97

Ascend.io

AUGUST 15, 2023

This same principle holds true in data management. You require a comprehensive solution that addresses every facet, from ingestion and transformation to orchestration and reverse ETL. Defense: Saving Money with Intelligent Data Refresh In football, a solid defense does more than just stop goals.

Data Management 52

Data Management Management Data Ingestion Data Pipeline 52

Rockset

AUGUST 4, 2021

With Snowflake, organizations get the simplicity of data management with the power of scaled-out data and distributed processing. Although Snowflake is great at querying massive amounts of data, the database still needs to ingest this data. Data ingestion must be performant to handle large amounts of data.

Data Ingestion 40

Data Ingestion Cloud Storage Data Warehouse Architecture 40

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. In fact, while only 3.5% That’s where our friends at Ascend.io

MongoDB 130

MongoDB MySQL Scala Machine Learning 130

phData: Data Engineering

FEBRUARY 21, 2025

A data warehouse acts as a single source of truth for an organization’s data, providing a unified view of its operations and enabling data-driven decision-making. A data warehouse enables advanced analytics, reporting, and business intelligence. Data integrations and pipelines can also impact latency.

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Data Engineering Podcast

JULY 23, 2023

In this episode he explains how investing in high performance and operationally simplified streaming with a familiar API can yield significant benefits for software and data teams together. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Building 130

Building Machine Learning SQL Python 130

Data Engineering Podcast

JUNE 12, 2022

In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. In fact, while only 3.5% In fact, while only 3.5%

Unstructured Data 130

Unstructured Data MongoDB MySQL Scala 130

Cloudera

FEBRUARY 8, 2021

Adding more wires and throwing more compute hardware to the problem is simply not viable considering the cost and complexities of today’s connected cars or the additional demands designed into electric cars (like battery management systems and eco-trip planning).

Manufacturing 108

Manufacturing Machine Learning Data Ingestion Electronics 108

Hevo

JUNE 26, 2024

This article gives information about Snowflake master data management, which you can use to enhance your business revenue. What is Master Data Management? Master data management (MDM) uses various tools and techniques to organize and structure master data in a standardized format.

Data Management 40

Data Management Management Data Ingestion Government 40

Striim

APRIL 18, 2025

Siloed storage : Critical business data is often locked away in disconnected databases, preventing a unified view. Delayed data ingestion : Batch processing delays insights, making real-time decision-making impossible. If data is delayed, outdated, or missing key details, leaders may act on the wrong assumptions.

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Snowflake

NOVEMBER 11, 2024

With Hybrid Tables’ fast, high-concurrency point operations, you can store application and workflow state directly in Snowflake, serve data without reverse ETL and build lightweight transactional apps while maintaining a single governance and security model for both transactional and analytical data — all on one platform.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Snowflake

APRIL 18, 2024

Legacy SIEM cost factors to keep in mind Data ingestion: Traditional SIEMs often impose limits to data ingestion and data retention. Snowflake allows security teams to store all their data in a single platform and maintain it all in a readily accessible state, with virtually unlimited cloud data storage capacity.

Data Lake 104

Data Lake Data Ingestion Bytes Cloud Computing 104

Rockset

MARCH 21, 2019

When we started Rockset, we envisioned building a powerful cloud data management system that was really easy to use. Making the data stack simpler is fundamental to making data usable by developers and data scientists. The data management should feel limitless.

SQL 52

SQL Data Management Management Engineering 52

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 130

Metadata MongoDB MySQL Scala 130

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing. See it in action and schedule a demo with one of our data experts today.

Data Management 52

Data Management Management Data Lake Data Governance 52

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing. See it in action and schedule a demo with one of our data experts today.

Data Management 52

Data Management Management Data Lake Data Governance 52

Cloudera

AUGUST 4, 2021

Using a scalable data management and analytics platform built on Cloudera Enterprise, Sikorsky can process and store data in a reliable way, and analyze full data sets across entire fleets. images, video, text, spectral data) or other input such as thermographic or acoustic signals. .

Unstructured Data 98

Unstructured Data Data Ingestion Government Machine Learning 98

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing.

Data Management 52

Data Management Management Data Lake Data Governance 52

Edureka

MAY 27, 2025

Data Governance Data Management Data Lineage Fabric allows users to track the origin and transformation path of any data asset by automatically tracking data movement across pipelines, transformations, and reports.

Architecture 52

Architecture BI Business Intelligence Raw Data 52

Cloudera

JANUARY 20, 2021

The data journey is not linear, but it is an infinite loop data lifecycle – initiating at the edge, weaving through a data platform, and resulting in business imperative insights applied to real business-critical problems that result in new data-led initiatives. Conclusion.

Manufacturing 110

Manufacturing Data Warehouse Kafka Retail 110

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares his journey from building a consumer product to launching a data pipeline service and how his frustrations as a product owner have informed his work at Hevo Data. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Data Pipeline 130

Data Pipeline Building MongoDB MySQL 130

Cloudera

NOVEMBER 1, 2023

The connector makes it easy to update the LLM context by loading, chunking, generating embeddings, and inserting them into the Pinecone database as soon as new data is available. High-level overview of real-time data ingest with Cloudera DataFlow to Pinecone vector database.

Machine Learning 110

Machine Learning Data Ingestion Database Architecture 110

Cloudera

APRIL 15, 2019

Cloudera Flow Management (CFM) is a no-code data ingestion and management solution powered by Apache NiFi. With a slick user interface, 300+ processors and the NiFi Registry, CFM delivers highly scalable data management and DevOps capabilities to the enterprise.

Management 17

Management Data Ingestion Data Collection Government 17

Cloudera

NOVEMBER 19, 2020

For many agencies, 80 percent of the work in support of anomaly detection and fraud prevention goes into routine tasks around data management. Inordinate time and effort are devoted to cleaning and preparing data, resulting in data bottlenecks that impede effective use of anomaly detection tools.

Government 119

Government Machine Learning Algorithm Cloud 119

Data Engineering Podcast

JULY 17, 2022

In this episode Joe Reis, founder of Ternary Data and co-author of "Fundamentals of Data Engineering", turns the tables and interviews the host, Tobias Macey, about his journey into podcasting, how he runs the show behind the scenes, and the other things that occupy his time. In fact, while only 3.5% Links Podcast.__init__

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

AUGUST 21, 2022

In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Lambda Architecture 100

Lambda Architecture MongoDB MySQL Scala 100

Cloudera

DECEMBER 14, 2021

But with growing demands, there’s a more nuanced need for enterprise-scale machine learning solutions and better data management systems. The 2021 Data Impact Awards aim to honor organizations who have shown exemplary work in this area. . For this, the RTA transformed its data ingestion and management processes. .

Transportation 111

Transportation Telecommunication Banking Data Lake 111

Cloudera

FEBRUARY 4, 2019

Cloudera DataFlow (CDF) is a scalable, real-time streaming data platform that collects, curates, and analyzes data so customers gain key insights for immediate actionable intelligence. CDF, as an end-to-end streaming data platform, emerges as a clear solution for managing data from the edge all the way to the enterprise.

Data Ingestion 73

Data Ingestion Retail Kafka Data Lake 73

Data Engineering Podcast

OCTOBER 12, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it?

Business Intelligence 100

Business Intelligence BI Consulting Data Ingestion 100

Data Engineering Podcast

MAY 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Database 100

Database Architecture Data Architecture PostgreSQL 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. In fact, while only 3.5% That’s where our friends at Ascend.io

Data Security 100

Data Security Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB MySQL Scala Data Lake 100

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 100

MongoDB MySQL Google Cloud Scala 100

Lyft Engineering

NOVEMBER 29, 2023

Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data. Druid enables low latency (real-time) data ingestion, flexible data exploration and fast data aggregation resulting in sub-second query latencies.

Kafka 109

Kafka Data Ingestion Architecture Datasets 109

Cloudera

APRIL 20, 2021

A modern streaming architecture consists of critical components that provide data ingestion, security and governance, and real-time analytics. The three fundamental parts of the architecture are: Data ingestion that acquires the data from different streaming sources and orchestrates and augments the data from other sources.

Kafka 95

Kafka Hospitality Retail Data Ingestion 95