Monte Carlo

NOVEMBER 22, 2024

At the front end, you’ve got your data ingestion layer —the workhorse that pulls in data from everywhere it lives. Once you’ve got the data flowing in, you need somewhere to put it. A pipeline has to be more than just functional, it has to be ready for growth and resilient to issues.

Data Engineering 52

Data Engineering Data Engineer Building Engineering 52

Data Engineering Podcast

JUNE 16, 2024

Your host is Tobias Macey and today I'm interviewing Kevin Liu about his use of Trino and Iceberg for Stripe's data lakehouse Interview Introduction How did you get involved in the area of data management? Can you describe what role Trino and Iceberg play in Stripe's data architecture?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Ascend.io

JULY 28, 2023

Data pipelines are integral to business operations, regardless of whether they are meticulously built in-house or assembled using various tools. As companies become more data-driven, the scope and complexity of data pipelines inevitably expand. Ready to fortify your data management practice?

Data Pipeline 52

Data Pipeline Architecture Lambda Architecture Data Architecture 52

Data Engineering Podcast

JUNE 17, 2023

Sign up free at dataengineeringpodcast.com/rudderstack - Your host is Tobias Macey and today I'm interviewing Satish Jayanthi about the practice and promise of building a column-aware data architecture through intentional modeling Interview Introduction How did you get involved in the area of data management?

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

The Modern Data Company

JULY 29, 2024

BCG research reveals a striking trend: the number of unique data vendors in large companies has nearly tripled over the past decade, growing from about 50 to 150. This dramatic increase in vendors hasn’t led to the expected data revolution. The limited reusability of data assets further exacerbates this agility challenge.

Data Architecture 52

Data Architecture Architecture Building Pipeline-centric 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

DataKitchen

NOVEMBER 5, 2024

This architecture is valuable for organizations dealing with large volumes of diverse data sources, where maintaining accuracy and accessibility at every stage is a priority. It sounds great, but how do you prove the data is correct at each layer? How do you ensure data quality in every layer ?

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Snowflake

JULY 27, 2023

Over the course of this journey, HomeToGo’s data needs have evolved considerably. After we had a successful trial period that checked all the boxes, we started our migration in autumn 2021 — together with moving all our data transformation management into the OSS version of dbt.

Data Architecture 52

Data Architecture Architecture Building Structured Data 52

Data Engineering Weekly

FEBRUARY 23, 2025

It allows different data platforms to access and share the same underlying data without copying, treating OTFs as a storage-layer abstraction. link] Sponsored: Webinar - The State of Airflow 2025 We asked 5,000+ data engineers how Airflow is shaping the modern DataOps landscape.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Netflix Tech

NOVEMBER 14, 2023

We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Furthermore, we’ll delve into the inner workings of Psyberg, its unique features, and how it integrates into our data pipelining workflows. This is mainly used to identify new changes since the last update.

Data Engineering 108

Data Engineering Data Engineer Engineering Metadata 108

Cloudera

DECEMBER 21, 2021

In working with thousands of customers deploying Spark applications, we saw significant challenges with managing Spark as well as automating, delivering, and optimizing secure data pipelines. We wanted to develop a service tailored to the data engineering practitioner built on top of a true enterprise hybrid data service platform.

Data Engineering 118

Data Engineering Data Engineer Engineering Data Pipeline 118

Data Engineering Weekly

SEPTEMBER 18, 2024

The advent of data lakes has changed the landscape of data infrastructure in two fundamental ways: 1. Decoupling of Storage and Compute : Data lakes allow observability tools to run alongside core data pipelines without competing for resources by separating storage from compute resources.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Data 98

Cloudera

MAY 10, 2024

Iceberg, a high-performance open-source format for huge analytic tables, delivers the reliability and simplicity of SQL tables to big data while allowing for multiple engines like Spark, Flink, Trino, Presto, Hive, and Impala to work with the same tables, all at the same time.

Government 103

Government Data Governance Big Data Data Pipeline 103

Monte Carlo

DECEMBER 30, 2024

Today, data quality isnt merely a business riskits an existential one. From a lack of necessary automation to a lack of incident management features, traditional data quality methods cant monitor all the ways your data pipelines can breakor help you resolve it quickly when they do. And thats a big problem for AI.

Architecture 52

Architecture Data Pipeline Data Engineering Data Engineer 52

Data Engineering Podcast

JULY 3, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB Data Warehouse 100

Data Engineering Podcast

OCTOBER 2, 2021

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. What are the driving factors for building a real-time data platform?

Building 100

Building BI Data Architecture Architecture 100

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. The communication between business units and data professionals is usually incomplete and inconsistent. Introduction to Data Mesh. Source: Thoughtworks.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Architecture 98

Data Engineering Podcast

JUNE 3, 2019

He also discusses how the abstractions provided by DataCoral allows his data scientists to remain productive without requiring dedicated data engineers. If you are either considering how to build a data pipeline or debating whether to migrate your existing ETL to a service this is definitely worth listening to for some perspective.

Media 100

Media Data Pipeline Machine Learning Data Science 100

Data Engineering Podcast

OCTOBER 28, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo! For someone who wants to get started with Dagster can you describe a typical workflow for writing a data pipeline?

Building 100

Building Data Pipeline Programming Language Kafka 100

Data Engineering Podcast

NOVEMBER 11, 2019

Even if you aren’t subject to specific rules regarding data protection it is definitely worth listening to get an overview of what you should be thinking about while building and running data pipelines. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo!

Designing 100

Designing Data Pipeline Programming Language Data 100

Cloudera

FEBRUARY 15, 2024

Seeing the future in a modern data architecture The key to successfully navigating these challenges lies in the adoption of a modern data architecture. The promise of a modern data architecture might seem like a distant reality, but we at Cloudera believe data can make what is impossible today, possible tomorrow.

Insurance 88

Insurance Banking Data Architecture Data Ingestion 88

The Modern Data Company

JANUARY 31, 2023

This post highlights exactly how our founders taught us to think differently about data and why it matters. Here are the cornerstones of this new paradigm: Data ownership is a construct Data pipelines should be accessible to everyone Data products should adapt to the organization, not vice versa.

Retail 98

Retail Healthcare Data Architecture Data Pipeline 98

DataKitchen

DECEMBER 30, 2020

CRN’s The 10 Hottest Data Science & Machine Learning Startups of 2020 (So Far). In June of 2020, CRN featured DataKitchen’s DataOps Platform for its ability to manage the data pipeline end-to-end combining concepts from Agile development, DevOps, and statistical process control: DataKitchen.

Manufacturing 96

Manufacturing Big Data Data Pipeline Data Science 96

Cloudera

FEBRUARY 22, 2022

Over the past decade, the successful deployment of large scale data platforms at our customers has acted as a big data flywheel driving demand to bring in even more data, apply more sophisticated analytics, and on-board many new data practitioners from business analysts to data scientists.

Metadata 110

Metadata Datasets BI SQL 110

Data Engineering Podcast

APRIL 3, 2022

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. What are the pitfalls in data architecture patterns that you commonly see organizations fall prey to?

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Cloudera

OCTOBER 19, 2023

While navigating so many simultaneous data-dependent transformations, they must balance the need to level up their data management practices—accelerating the rate at which they ingest, manage, prepare, and analyze data—with that of governing this data.

Telecommunication 76

Telecommunication Data Architecture Government Architecture 76

Cloudera

SEPTEMBER 26, 2023

As lakehouse architectures (including offerings from Cloudera and IBM) become the norm for data processing and building AI applications, a robust streaming service becomes a critical building block for modern data architectures. Apache Kafka has evolved into the most widely-used streaming platform, capable of ingesting and processing (..)

Kafka 95

Kafka Technology IT Government 95

AltexSoft

FEBRUARY 11, 2023

To get a better understanding of a data architect’s role, let’s clear up what data architecture is. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. What is the main difference between a data architect and a data engineer?

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Podcast

JULY 28, 2019

Rather than manually defining all of the mappings ahead of time, we can rely on the power of graph databases and some strategic metadata to allow connections to occur as the data becomes available. If you are struggling to maintain a tangle of data pipelines then you might find some new ideas for reducing your workload.

Data Integration 100

Data Integration Metadata Architecture Media 100

Confluent

SEPTEMBER 19, 2019

But I follow it up quickly with a second and potentially unrelated pattern: real-time data pipelines. Batch vs. real-time streams of data. So, businesses need data-driven insights based on things that are happening right now, and that’s where real-time data pipelines come in.

Kafka 14

Kafka Data Pipeline Bytes Data Architect 14

DataKitchen

SEPTEMBER 17, 2021

A data mesh implemented on a DataOps process hub, like the DataKitchen Platform, can avoid the bottlenecks characteristic of large, monolithic enterprise data architectures. Doing so will give you the agility that your data organization needs to cope with new analytics requirements. Conclusion.

Architecture 94

Architecture Data Architecture Government Raw Data 94

DataKitchen

NOVEMBER 29, 2021

Data Gets Meshier. 2022 will bring further momentum behind modular enterprise architectures like data mesh. The data mesh addresses the problems characteristic of large, complex, monolithic data architectures by dividing the system into discrete domains managed by smaller, cross-functional teams.

Data Lake 98

Data Lake Manufacturing Architecture Data Architecture 98

Snowflake

DECEMBER 12, 2023

Iceberg Tables bring the easy management and great performance of Snowflake to data stored externally in an open source format. Data Pipelines Improved processing for streaming data with Dynamic Tables – public preview Streaming and CDC data can be challenging to handle. The retention period will be 1 year.

Metadata 114

Metadata Python AWS Government 114

Confluent

JULY 23, 2019

She has 15 years of experience working with code and customers to build scalable data architectures, integrating relational and big data technologies. Gwen is the author of “Kafka—The Definitive Guide” and “Hadoop Application Architectures,” and a frequent presenter at industry conferences.

Kafka 18

Kafka Hadoop Media Software Engineering 18

phData: Data Engineering

FEBRUARY 21, 2025

Further, choosing the right CSP subscription model can help an organization meet its SLAs and data availability requirements. Security For most organizations, security is a top priority when establishing a data architecture. Organizations want to ensure that their data is secure both at rest and in-transit.

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52