Data, Data Lake and Data Pipeline - Data Engineering Digest

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Cliff Crosland co-founded Scanner to provide fast querying of high scale log data for security auditing. SIEM) A query engine is useless without data to analyze.

Data Lake

Data Lake Building High Quality Data AWS

Keep Your Data Lake Fresh With Real Time Streams Using Estuary

Data Engineering Podcast

MAY 21, 2023

Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.

Data Lake

Data Lake Machine Learning Kafka Data Warehouse

A Guide to Data Pipelines (And How to Design One From Scratch)

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. That’s where real-time data, and stream processing can help. We’ll answer the question, “What are data pipelines?”

Data Pipeline

Data Pipeline Designing Data Lake Data Warehouse

Webinars

Prepare Now: 2025s Must-Know Trends For Product And Data Leaders

MORE WEBINARS

What is a Data Pipeline (and 7 Must-Have Features of Modern Data Pipelines)

Striim

OCTOBER 11, 2024

A well-executed data pipeline can make or break your company’s ability to leverage real-time insights and stay competitive. Thriving in today’s world requires building modern data pipelines that make moving data and extracting valuable insights quick and simple. What is a Data Pipeline?

Data Pipeline

Data Pipeline MongoDB Unstructured Data Data Lake

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Charting A Path For Streaming Data To Fill Your Data Lake With Hudi

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake

Data Lake Data Warehouse Hadoop Architecture

CI/CD for Data Pipelines: A Game-Changer with AnalyticsCreator

Data Science Blog: Data Engineering

MAY 20, 2024

Continuous Integration and Continuous Delivery (CI/CD) for Data Pipelines: It is a Game-Changer with AnalyticsCreator! The need for efficient and reliable data pipelines is paramount in data science and data engineering. They transform data into a consistent format for users to consume.

Data Pipeline

Data Pipeline BI Data Lake Data Warehouse

Exploring Processing Patterns For Streaming Data Integration In Your Data Lake

Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset.

Data Lake

Data Lake Data Integration Lambda Architecture Process

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Data Pipeline

Data Pipeline Building MongoDB Scala

Tips to Build a Robust Data Lake Infrastructure

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake

Data Lake Building Raw Data ETL Tools

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL

SQL Data Lake High Quality Data Data Pipeline

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action? What problems are you trying to solve with Dagster+?

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Being Data Driven At Stripe With Trino And Iceberg

Data Engineering Podcast

JUNE 16, 2024

Summary Stripe is a company that relies on data to power their products and business. In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform.

Data Lake

Data Lake High Quality Data Metadata Machine Learning

Insights And Advice On Building A Data Lake Platform From Someone Who Learned The Hard Way

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Struggling with broken pipelines? Missing data? Stale dashboards?

Data Lake

Data Lake Building Architecture BI

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

Summary Sharing data is a simple concept, but complicated to implement well. There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data Government Data Pipeline

Streaming Data Pipelines Made SQL With Decodable

Data Engineering Podcast

OCTOBER 28, 2021

Summary Streaming data systems have been growing more capable and flexible over the past few years. Despite this, it is still challenging to build reliable pipelines for stream processing. Struggling with broken pipelines? Missing data? Start trusting your data with Monte Carlo today! Stale dashboards?

Data Pipeline

Data Pipeline SQL Data Warehouse Data Lake

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Data Lake

Data Lake High Quality Data Architecture Data Pipeline

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. Different roles and tasks in the business need their own ways to access and analyze the data in the organization. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data BI Data Workflow

Zenlytic Is Building You A Better Coworker With AI Agents

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Data lakes are notoriously complex.

Building

Building Data Lake High Quality Data Business Intelligence

Evaluating Data Observability Tools: A Comprehensive Guide

Data Engineering Weekly

SEPTEMBER 18, 2024

Click the Link Below to Get Your Free Data Observability Buyers Guide: [link] The Buyer Guide for Data Observability is out. I w ant to extend my gratitude to the Data Heroes Community for their valuable insights and discussions, which served as the foundation for this piece.

Data Lake

Data Lake Data Pipeline Unstructured Data Datasets

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database

Database Data Lake High Quality Data Data Workflow

Improve Data Quality Through Engineering Rigor And Business Engagement With Synq

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems. Want to see Starburst in action?

Pipeline-centric

Pipeline-centric Engineering Data Lake High Quality Data

Data Warehouse vs Data Lake vs Data Lakehouse: Definitions, Similarities, and Differences

Monte Carlo

AUGUST 25, 2023

Modern companies are ingesting, storing, transforming, and leveraging more data to drive more decision-making than ever before. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake

Data Lake Data Warehouse Unstructured Data Raw Data

Release Management For Data Platform Services And Logic

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex.

Management

Management Data Lake High Quality Data Machine Learning

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

Modern Customer Data Platform Principles

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Want to see Starburst in action?

Data Lake

Data Lake High Quality Data NoSQL Data Warehouse

Stitching Together Enterprise Analytics With Microsoft Fabric

Data Engineering Podcast

JUNE 23, 2024

Summary Data lakehouse architectures have been gaining significant adoption. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. What are the benefits of embedding Copilot into the data engine?

Data Lake

Data Lake High Quality Data Hadoop Machine Learning

Data Pipeline Architecture Explained: 6 Diagrams and Best Practices

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline

Data Pipeline Architecture Data Lake Data Warehouse

Practical First Steps In Data Governance For Long Term Success

Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. What are some of the misconceptions that you encounter about data governance?

Data Governance

Data Governance Government Data Lake High Quality Data

How to Build a Data Pipeline in 6 Steps

Ascend.io

JANUARY 2, 2024

Getting your hands on the right data at the right time is the lifeblood of any forward-thinking company. But let’s be honest, creating effective, robust, and reliable data pipelines, the ones that feed your company’s reporting and analytics, is no walk in the park. What Is a Data Pipeline?

Data Pipeline

Data Pipeline Building Raw Data Data Warehouse

Data Lake vs. Data Warehouse: Differences and Similarities

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. Structuring data refers to converting unstructured data into tables and defining data types and relationships based on a schema. What is Data Warehouse? . What is Data Lake? .

Data Lake

Data Lake Data Warehouse Unstructured Data Amazon Web Services

Automated Data Pipelines: What You Need to Know

Ascend.io

AUGUST 22, 2023

Businesses are in a continuous race to harness data as a powerful propellant for innovation. Yet, the task of efficient and prompt data delivery remains a challenge. The demands of building, scaling, and maintaining data pipelines have grown increasingly complex and error-prone. What Are Automated Data Pipelines?

Data Pipeline

Data Pipeline Raw Data Bytes Transportation

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. What are the open questions today in technical scalability of data engines?

Data Process

Data Process Process Data Lake High Quality Data

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineer

Data Engineer Data Engineering Python Engineering

Why Modernizing the First Mile of the Data Pipeline Can Accelerate all Analytics

Cloudera

AUGUST 13, 2021

Every enterprise is trying to collect and analyze data to get better insights into their business. Whether it is consuming log files, sensor metrics, and other unstructured data, most enterprises manage and deliver data to the data lake and leverage various applications like ETL tools, search engines, and databases for analysis.

Data Pipeline

Data Pipeline Data Lake ETL Tools Unstructured Data

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Building

Building Data Lake High Quality Data Machine Learning

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Building

Building Data Lake High Quality Data Machine Learning

Designing Data Transfer Systems That Scale

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Materialize, you can!

Systems

Systems Designing Data Lake SQL

What is AWS Data Pipeline?

ProjectPro

JUNE 16, 2022

An AWS data pipeline helps businesses move and unify their data to support several data-driven initiatives. It enables flow from a data lake to an analytics database or an application to a data warehouse. This blog will teach you about AWS Data Pipeline, its architecture, components, and benefits.

Data Pipeline

Data Pipeline AWS Amazon Web Services Data Consolidation

Data Lake Explained: A Comprehensive Guide to Its Architecture and Use Cases

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake

Data Lake Architecture IT Amazon Web Services

Designing Data Platforms For Fintech Companies

Data Engineering Podcast

DECEMBER 31, 2023

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Want to see Starburst in action?

Designing

Designing Data Lake High Quality Data SQL

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Kafka

Kafka Data Lake High Quality Data SQL

Addressing The Challenges Of Component Integration In Data Platform Architectures

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Architecture

Architecture Data Lake High Quality Data SQL

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database

Non-relational Database Relational Database Database Designing

Build A Data Lake For Your Security Logs With Scanner

Keep Your Data Lake Fresh With Real Time Streams Using Estuary

Webinars

Trending Sources

A Guide to Data Pipelines (And How to Design One From Scratch)

Webinars

What is a Data Pipeline (and 7 Must-Have Features of Modern Data Pipelines)

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Charting A Path For Streaming Data To Fill Your Data Lake With Hudi

CI/CD for Data Pipelines: A Game-Changer with AnalyticsCreator

Exploring Processing Patterns For Streaming Data Integration In Your Data Lake

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Tips to Build a Robust Data Lake Infrastructure

Tackling Real Time Streaming Data With SQL Using RisingWave

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Being Data Driven At Stripe With Trino And Iceberg

Insights And Advice On Building A Data Lake Platform From Someone Who Learned The Hard Way

Data Sharing Across Business And Platform Boundaries

Streaming Data Pipelines Made SQL With Decodable

Version Your Data Lakehouse Like Your Software With Nessie

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Zenlytic Is Building You A Better Coworker With AI Agents

Evaluating Data Observability Tools: A Comprehensive Guide

Reconciling The Data In Your Databases With Datafold

Improve Data Quality Through Engineering Rigor And Business Engagement With Synq

Data Warehouse vs Data Lake vs Data Lakehouse: Definitions, Similarities, and Differences

Release Management For Data Platform Services And Logic

Making Email Better With AI At Shortwave

Modern Customer Data Platform Principles

Stitching Together Enterprise Analytics With Microsoft Fabric

Data Pipeline Architecture Explained: 6 Diagrams and Best Practices

Practical First Steps In Data Governance For Long Term Success

How to Build a Data Pipeline in 6 Steps

Data Lake vs. Data Warehouse: Differences and Similarities

Automated Data Pipelines: What You Need to Know

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Seamless SQL And Python Transformations For Data Engineers And Analysts With SQLMesh

Why Modernizing the First Mile of the Data Pipeline Can Accelerate all Analytics

Build Your Second Brain One Piece At A Time

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Designing Data Transfer Systems That Scale

What is AWS Data Pipeline?

Data Lake Explained: A Comprehensive Guide to Its Architecture and Use Cases

Designing Data Platforms For Fintech Companies

Troubleshooting Kafka In Production

Addressing The Challenges Of Component Integration In Data Platform Architectures

Designing A Non-Relational Database Engine

Stay Connected