Striim

AUGUST 14, 2024

When implemented effectively, smart data pipelines seamlessly integrate data from diverse sources, enabling swift analysis and actionable insights. They empower data analysts and business users alike by providing critical information while protecting sensitive production systems. What is a Smart Data Pipeline?

Data Pipeline 52

Data Pipeline Data Architecture Transportation 52

Striim

APRIL 18, 2025

How Organizations Can Overcome Data Quality and Availability Challenges Many businesses are shifting toward real-time data pipelines to ensure their AI and analytics strategies are built on reliable information. Enabling AI & ML with Adaptive Data Pipelines AI models require ongoing updates to stay relevant.

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Data Engineering Podcast

JUNE 16, 2024

Your host is Tobias Macey and today I'm interviewing Kevin Liu about his use of Trino and Iceberg for Stripe's data lakehouse Interview Introduction How did you get involved in the area of data management? Can you describe what role Trino and Iceberg play in Stripe's data architecture?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Ascend.io

JULY 28, 2023

Data pipelines are integral to business operations, regardless of whether they are meticulously built in-house or assembled using various tools. As companies become more data-driven, the scope and complexity of data pipelines inevitably expand. Ready to fortify your data management practice?

Data Pipeline 52

Data Pipeline Architecture Lambda Architecture Data Architecture 52

The Modern Data Company

JULY 29, 2024

BCG research reveals a striking trend: the number of unique data vendors in large companies has nearly tripled over the past decade, growing from about 50 to 150. This dramatic increase in vendors hasn’t led to the expected data revolution. The limited reusability of data assets further exacerbates this agility challenge.

Data Architecture 52

Data Architecture Architecture Building Pipeline-centric 52

Data Engineering Podcast

JUNE 17, 2023

Sign up free at dataengineeringpodcast.com/rudderstack - Your host is Tobias Macey and today I'm interviewing Satish Jayanthi about the practice and promise of building a column-aware data architecture through intentional modeling Interview Introduction How did you get involved in the area of data management?

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

DataKitchen

NOVEMBER 5, 2024

This architecture is valuable for organizations dealing with large volumes of diverse data sources, where maintaining accuracy and accessibility at every stage is a priority. It sounds great, but how do you prove the data is correct at each layer? How do you ensure data quality in every layer ?

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Weekly

FEBRUARY 23, 2025

It allows different data platforms to access and share the same underlying data without copying, treating OTFs as a storage-layer abstraction. link] Sponsored: Webinar - The State of Airflow 2025 We asked 5,000+ data engineers how Airflow is shaping the modern DataOps landscape.

Data Engineering 63

Data Engineering Data Engineer Engineering Kafka 63

Snowflake

JULY 27, 2023

Over the course of this journey, HomeToGo’s data needs have evolved considerably. After we had a successful trial period that checked all the boxes, we started our migration in autumn 2021 — together with moving all our data transformation management into the OSS version of dbt.

Data Architecture 52

Data Architecture Architecture Building Structured Data 52

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Edureka

APRIL 22, 2025

Its multi-cluster shared data architecture is one of its primary features. Additionally, Fabric has deep integrations with Power BI for visualization and Microsoft Purview for governance, resulting in a smooth experience for both business users and data professionals.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Netflix Tech

NOVEMBER 14, 2023

We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Furthermore, we’ll delve into the inner workings of Psyberg, its unique features, and how it integrates into our data pipelining workflows. This is mainly used to identify new changes since the last update.

Data Engineering 108

Data Engineering Data Engineer Engineering Metadata 108

Cloudera

DECEMBER 21, 2021

In working with thousands of customers deploying Spark applications, we saw significant challenges with managing Spark as well as automating, delivering, and optimizing secure data pipelines. We wanted to develop a service tailored to the data engineering practitioner built on top of a true enterprise hybrid data service platform.

Data Engineering 118

Data Engineering Data Engineer Engineering Data Pipeline 118

Data Engineering Weekly

SEPTEMBER 18, 2024

The advent of data lakes has changed the landscape of data infrastructure in two fundamental ways: 1. Decoupling of Storage and Compute : Data lakes allow observability tools to run alongside core data pipelines without competing for resources by separating storage from compute resources.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Data 98

Cloudera

MAY 10, 2024

Iceberg, a high-performance open-source format for huge analytic tables, delivers the reliability and simplicity of SQL tables to big data while allowing for multiple engines like Spark, Flink, Trino, Presto, Hive, and Impala to work with the same tables, all at the same time.

Government 103

Government Data Governance Big Data Data Pipeline 103

Monte Carlo

DECEMBER 30, 2024

Today, data quality isnt merely a business riskits an existential one. From a lack of necessary automation to a lack of incident management features, traditional data quality methods cant monitor all the ways your data pipelines can breakor help you resolve it quickly when they do. And thats a big problem for AI.

Architecture 52

Architecture Data Pipeline Data Engineering Data Engineer 52

Data Engineering Podcast

JULY 3, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 2, 2021

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. What are the driving factors for building a real-time data platform?

Building 100

Building BI Data Architecture Architecture 100

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. The communication between business units and data professionals is usually incomplete and inconsistent. Introduction to Data Mesh. Source: Thoughtworks.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Architecture 98

Cloudera

FEBRUARY 14, 2022

It’s no longer driven by data volumes, but containerization, separation of storage and compute, and democratization of analytics. The same key tenants powering DE in the public clouds are now available in the data center. Data Engineering should not be limited by one cloud vendor or data locality.

Data Engineering 113

Data Engineering Data Engineer Engineering Data Pipeline 113

Data Engineering Podcast

JUNE 3, 2019

He also discusses how the abstractions provided by DataCoral allows his data scientists to remain productive without requiring dedicated data engineers. If you are either considering how to build a data pipeline or debating whether to migrate your existing ETL to a service this is definitely worth listening to for some perspective.

Media 100

Media Data Pipeline Machine Learning Data Science 100

Data Engineering Podcast

OCTOBER 28, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo! For someone who wants to get started with Dagster can you describe a typical workflow for writing a data pipeline?

Building 100

Building Data Pipeline Programming Language Kafka 100

Data Engineering Podcast

NOVEMBER 11, 2019

Even if you aren’t subject to specific rules regarding data protection it is definitely worth listening to get an overview of what you should be thinking about while building and running data pipelines. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo!

Designing 100

Designing Data Pipeline Programming Language Data 100

Cloudera

FEBRUARY 15, 2024

Seeing the future in a modern data architecture The key to successfully navigating these challenges lies in the adoption of a modern data architecture. The promise of a modern data architecture might seem like a distant reality, but we at Cloudera believe data can make what is impossible today, possible tomorrow.

Insurance 88

Insurance Banking Data Architecture Data Ingestion 88

DataKitchen

DECEMBER 30, 2020

CRN’s The 10 Hottest Data Science & Machine Learning Startups of 2020 (So Far). In June of 2020, CRN featured DataKitchen’s DataOps Platform for its ability to manage the data pipeline end-to-end combining concepts from Agile development, DevOps, and statistical process control: DataKitchen.

Manufacturing 96

Manufacturing Big Data Data Pipeline Data Science 96

Cloudera

FEBRUARY 22, 2022

Over the past decade, the successful deployment of large scale data platforms at our customers has acted as a big data flywheel driving demand to bring in even more data, apply more sophisticated analytics, and on-board many new data practitioners from business analysts to data scientists.

Metadata 110

Metadata Datasets BI SQL 110

Edureka

APRIL 9, 2025

Development of Some Relevant Skills and Knowledge Data Engineering Fundamentals: Theoretical knowledge of data loading patterns, data architectures, and orchestration processes. Data Analytics: Capability to effectively use tools and techniques for analyzing data and drawing insights.

Engineering 40

Engineering Data Ingestion Data Lake Programming Language 40

Data Engineering Podcast

APRIL 3, 2022

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. What are the pitfalls in data architecture patterns that you commonly see organizations fall prey to?

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Cloudera

OCTOBER 19, 2023

While navigating so many simultaneous data-dependent transformations, they must balance the need to level up their data management practices—accelerating the rate at which they ingest, manage, prepare, and analyze data—with that of governing this data.

Telecommunication 76

Telecommunication Data Architecture Government Architecture 76

Cloudera

SEPTEMBER 26, 2023

As lakehouse architectures (including offerings from Cloudera and IBM) become the norm for data processing and building AI applications, a robust streaming service becomes a critical building block for modern data architectures. Apache Kafka has evolved into the most widely-used streaming platform, capable of ingesting and processing (..)

Kafka 95

Kafka Technology IT Government 95

AltexSoft

FEBRUARY 11, 2023

To get a better understanding of a data architect’s role, let’s clear up what data architecture is. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. What is the main difference between a data architect and a data engineer?

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Podcast

JULY 28, 2019

Rather than manually defining all of the mappings ahead of time, we can rely on the power of graph databases and some strategic metadata to allow connections to occur as the data becomes available. If you are struggling to maintain a tangle of data pipelines then you might find some new ideas for reducing your workload.

Data Integration 100

Data Integration Metadata Architecture Media 100

Confluent

SEPTEMBER 19, 2019

But I follow it up quickly with a second and potentially unrelated pattern: real-time data pipelines. Batch vs. real-time streams of data. So, businesses need data-driven insights based on things that are happening right now, and that’s where real-time data pipelines come in.

Kafka 14

Kafka Data Pipeline Bytes Data Architect 14

DataKitchen

SEPTEMBER 17, 2021

A data mesh implemented on a DataOps process hub, like the DataKitchen Platform, can avoid the bottlenecks characteristic of large, monolithic enterprise data architectures. Doing so will give you the agility that your data organization needs to cope with new analytics requirements. Conclusion.

Architecture 94

Architecture Data Architecture Government Raw Data 94

Snowflake

DECEMBER 12, 2023

Iceberg Tables bring the easy management and great performance of Snowflake to data stored externally in an open source format. Data Pipelines Improved processing for streaming data with Dynamic Tables – public preview Streaming and CDC data can be challenging to handle. The retention period will be 1 year.

Metadata 118

Metadata Python AWS Government 118