phData: Data Engineering

NOVEMBER 8, 2024

Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms. In this blog, we will discuss: What is the Open Table format (OTF)? First, we create an Iceberg table in Snowflake and then insert some data.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Netflix Tech

DECEMBER 3, 2022

Its main purpose is to enable easy unit testing of your data pipelines, but it can technically be used in any other situations as a readable data format for small data sets. All the above commands are very likely to be described in separate future blog posts, but right now let’s focus on the dataflow sample command.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Snowflake

AUGUST 7, 2024

As we describe in this blog post , the top-k feature uses runtime information — namely, the current contents of the top-k elements — to skip micro-partitions where we can guarantee that they won’t contribute to the overall result. Snowflake starts processing those partitions first. on average, with some queries also reaching up to 99.8%

Metadata 109

Metadata Algorithm Process Utilities 109

Netflix Tech

NOVEMBER 14, 2023

In this context, managing the data, especially when it arrives late, can present a substantial challenge! In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Let’s dive in! To solve these problems, we came up with Psyberg!

Data Engineering 108

Data Engineering Data Engineer Engineering Metadata 108

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Data and Metadata: Data inputs and data outputs produced based on the application logic.

Architecture 124

Architecture Metadata Kafka Government 124

Pinterest Engineering

NOVEMBER 18, 2024

Liang Mou; Staff Software Engineer, Logging Platform | Elizabeth (Vi) Nguyen; Software Engineer I, Logging Platform | In today’s data-driven world, businesses need to process and analyze data in real-time to make informed decisions. What is Change Data Capture? Why is CDC Important? or its affiliates.

Kafka 50

Kafka MySQL Database Software Engineering 50

Ascend.io

DECEMBER 4, 2024

Automation, AI, DataOps, and strategic alignment are no longer optional —they are essential components of a successful data strategy. As we look towards 2025, it’s clear that data teams must evolve to meet the demands of evolving technology and opportunities. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing.

Metadata 101

Metadata Data Pipeline Scala Data Process 101

Cloudera

AUGUST 26, 2021

In addition to big data workloads, Ozone is also fully integrated with authorization and data governance providers namely Apache Ranger & Apache Atlas in the CDP stack. While we walk through the steps one by one from data ingestion to analysis, we will also demonstrate how Ozone can serve as an ‘S3’ compatible object store.

Data Science 113

Data Science Cloud Hadoop Metadata 113

Data Engineering Weekly

MARCH 23, 2025

[link] Georg Heiler: Upskilling data engineers What should I prefer for 2028, or how can I break into data engineering? I honestly don’t have a solid answer, but this blog is an excellent overview of upskilling. These are common LinkedIn requests.

Data Engineering 63

Data Engineering Data Engineer Engineering Data 63

Data Engineering Weekly

JUNE 24, 2024

link] Netflix: A Recap of the Data Engineering Open Forum at Netflix Netflix publishes a recap of all the talks in the first Data Engineering open forum tech meetups. The blog contains a summary of each talk and a link to the YouTube channel with all the talks. Are there enough usecases?

Data Engineering 110

Data Engineering Data Engineer Engineering Google Cloud 110

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. This nuanced integration of data and technology empowers us to offer bespoke content recommendations.

Kafka 66

Kafka Datasets Metadata Utilities 66

Maxime Beauchemin

JANUARY 20, 2017

The fact that ETL tools evolved to expose graphical interfaces seems like a detour in the history of data processing, and would certainly make for an interesting blog post of its own. Sure, there’s a need to abstract the complexity of data processing, computation and storage.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

JANUARY 12, 2025

This blog captures the current state of Agent adoption, emerging software engineering roles, and the use case category. Generative AI demands the processing of vast amounts of diverse, unstructured data (e.g., meeting recordings and videos), which contrasts with traditional SQL-centric systems for structured data.

Pipeline-centric 58

Pipeline-centric Data Engineering Data Engineer Engineering 58

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. You can read part 1, here: Digital Transformation is a Data Journey From Edge to Insight. The first blog introduced a mock connected vehicle manufacturing company, The Electric Car Company (ECC), to illustrate the manufacturing data path through the data lifecycle.

Data Pipeline 118

Data Pipeline Building Manufacturing Data Warehouse 118

Data Engineering Weekly

MAY 16, 2023

In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive? I won’t bore you with the importance of data quality in the blog. Let’s talk about the data processing types.

Engineering 111

Engineering Kafka Data Pipeline Data Warehouse 111

Cloudera

MARCH 3, 2022

Data users in these enterprises don’t know how data is derived and lack confidence in whether it’s the right source to use. . If data access policies and lineage aren’t consistent across an organization’s private cloud and public clouds, gaps will exist in audit logs. From Bad to Worse.

Data Governance 100

Data Governance Government Metadata Medical 100

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. Iceberg is a high-performance open table format for huge analytic data sets. The post Streaming Ingestion for Apache Iceberg With Cloudera Stream Processing appeared first on Cloudera Blog.

Process 114

Process SQL Kafka Database 114

Data Engineering Weekly

DECEMBER 10, 2023

The blog is an excellent comparison study of Ray vs. Dask’s performance. The author discusses the OneTable sync mechanism among all three major LakeHouse formats in this blog. The blog discusses Psyberg’s two operational models, stateless & stateful data processing. Stores metadata to utilize later.

Data Engineering 77

Data Engineering Data Engineer Engineering Metadata 77

Cloudera

JULY 19, 2023

It is a replicated, highly-available service that is responsible for managing the metadata for all objects stored in Ozone. As Ozone scales to exabytes of data, it is important to ensure that Ozone Manager can perform at scale. The hardware specifications are included at the end of this blog.

Management 70

Management Metadata Datasets Architecture 70

Pinterest Engineering

OCTOBER 11, 2024

In Part 2 of our blog series, we described how we were able to integrate Ray(™) into our existing ML infrastructure. In this blog post, we will discuss a second type of popular application of Ray(™) at Pinterest: offline batch inference of ML models. Ray Data is not bound to any specific ML library.

Datasets 71

Datasets Software Engineer Software Engineering Metadata 71

Cloudera

DECEMBER 15, 2022

With in-place table migration, you can rapidly convert to Iceberg tables since there is no need to regenerate data files. Only metadata will be regenerated. Newly generated metadata will then point to source data files as illustrated in the diagram below. . Data quality using table rollback. Metadata management .

Cloud 80

Cloud Metadata Data Warehouse Google Cloud 80

Cloudera

MARCH 23, 2022

The table information (such as schema, partition) is stored as part of the metadata (manifest) file separately, making it easier for applications to quickly integrate with the tables and the storage formats of their choice. Change data capture (CDC). 3: Open Performance.

Metadata 92

Metadata Data Architecture Machine Learning BI 92

Workfall

JUNE 12, 2023

So, embrace the power of Change Data Capture, and embark on a captivating journey where the magic of real-time data awaits. In this blog, we will cover: What Is CDC and Its Benefits? These additional columns store metadata like timestamps, user IDs, and change types, ensuring granular change tracking and auditability.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

Striim

MARCH 21, 2025

During the transformation phase, data is processed and converted into the appropriate format for the target destination. While legacy ETL has a slow transformation step, modern ETL platforms replace disk-based processing with in-memory processing to allow for real-time data processing, enrichment, and analysis.

IT 52

IT Data Lake Data Warehouse Relational Database 52

Netflix Tech

OCTOBER 19, 2020

In our previous blog post we introduced Edgar, our troubleshooting tool for streaming sessions. We could also get contextual information about the streaming session by joining relevant traces with account metadata and service logs. The next challenge was to stream large amounts of traces via a scalable data processing platform.

Building 95

Building Transportation Java Metadata 95

Cloudera

APRIL 1, 2024

We scored the highest in hybrid, intercloud, and multi-cloud capabilities because we are the only vendor in the market with a true hybrid data platform that can run on any cloud including private cloud to deliver a seamless, unified experience for all data, wherever it lies. Sign up for a trial to see for yourself.

Cloud 105

Cloud Unstructured Data Metadata Government 105

Cloudera

AUGUST 16, 2021

CDF-PC enables Apache NiFi users to run their existing data flows on a managed, auto-scaling platform with a streamlined way to deploy NiFi data flows and a central monitoring dashboard making it easier than ever before to operate NiFi data flows at scale in the public cloud. The need for a cloud-native Apache NiFi service.

Cloud 122

Cloud Unstructured Data Utilities Metadata 122

Data Engineering Podcast

JUNE 29, 2020

This was a great conversation about the complexities of working in a niche domain of data analysis and how to build a pipeline of high quality data from collection to analysis.

Data Collection 100

Data Collection Management High Quality Data Metadata 100

Netflix Tech

DECEMBER 21, 2020

On the other hand, these optimizations themselves need to be sufficiently inexpensive to justify their own processing cost over the gains they bring. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Data Warehouse 84

Data Warehouse Metadata Algorithm Data 84

Christophe Blefari

JANUARY 20, 2024

He wrote some years ago 3 articles defining data engineering field. Some concepts When doing data engineering you can touch a lot of different concepts. Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). Is it really modern?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

LinkedIn Engineering

JUNE 15, 2023

In this blog post, we will discuss the AvroTensorDataset API, techniques we used to improve data processing speeds by up to 162x over existing solutions (thereby decreasing overall training time by up to 66%), and performance results from benchmarks and production. The rest of the steps act as consumers of prefetched bytes.

Datasets 102

Datasets Bytes Process Data Ingestion 102

Cloudera

JUNE 18, 2022

The first generation of the Hive Metastore attempted to address the performance considerations to run SQL efficiently on a data lake. It provided the concept of a database, schemas, and tables for describing the structure of a data lake in a way that let BI tools traverse the data efficiently.

Data Lake 94

Data Lake Data Warehouse BI SQL 94

Cloudera

JUNE 16, 2022

Use cases like fraud detection, network threat analysis, manufacturing intelligence, commerce optimization, real-time offers, instantaneous loan approvals, and more are now possible by moving the data processing components up the stream to address these real-time needs. . Not in the manufacturing space? Not to worry.

Kafka 88

Kafka Manufacturing Data Lake SQL 88

Databand.ai

AUGUST 30, 2023

Data Performance Testing Data performance testing is the process of evaluating the efficiency, effectiveness, and scalability of your data processing systems and infrastructure. To perform data performance testing, you should first establish performance benchmarks and targets for your data processing systems.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52