Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. How is everyone going to find the data they need, and understand it?

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Weekly

NOVEMBER 24, 2024

The blog highlights how moving from 6-character base-64 to 20-digit base-2 file distribution brings more distribution in S3 and reduces request failures. The blog is a good summary of how to use Snowflake QUERY_TAG to measure and monitor query performance.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata BI Data Lake Business Intelligence 84

Confluent

AUGUST 1, 2019

The Kafka Streams API boasts a number of capabilities that make it well suited for maintaining the global state of a distributed system. At Imperva, we took advantage of Kafka Streams to build shared state microservices that serve as fault-tolerant, highly available single sources of truth about the state of objects in our system.

Kafka 20

Kafka Systems Building Metadata 20

Cloudera

JANUARY 26, 2024

This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control. First, a set of initial metadata objects are created by the data steward.

Metadata 73

Metadata Data Governance Government Management 73

Cloudera

SEPTEMBER 15, 2022

In this blog post, we will talk about a single Ozone cluster with the capabilities of both Hadoop Core File System (HCFS) and Object Store (like Amazon S3). A unified storage architecture that can store both files and objects and provide a flexible, scalable, and high-performance system. Bucket types. release version.

Systems 87

Systems Hadoop Metadata Telecommunication 87

Data Engineering Weekly

MARCH 5, 2025

This blog post expands on that insightful conversation, offering a critical look at Iceberg's potential and the hurdles organizations face when adopting it. Hadoop achieved this through distributed processing and storage, using a framework called MapReduce and the Hadoop Distributed File System (HDFS). Trino, Spark, Snowflake, DuckDB).

Hadoop 57

Hadoop Metadata Data Ingestion Data Governance 57

Pinterest Engineering

APRIL 4, 2025

In this blog, we will go through the technical design and share some offline and online results for our LLM-based search relevance pipeline. Pin Text Representations Pins on Pinterest are rich multimedia entities that feature images, videos, and other contents, often linked to external webpages or blogs.

Machine Learning 45

Machine Learning Metadata Architecture Datasets 45

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. Imagine a library with millions of books but no catalog system to organize them. This is what managing data without metadata feels like. What is Metadata? Chaos, right?

Metadata 52

Metadata IT Government High Quality Data 52

Data Engineering Weekly

NOVEMBER 3, 2024

Foundation Capital: A System of Agents brings Service-as-Software to life software is no longer simply a tool for organizing work; software becomes the worker itself, capable of understanding, executing, and improving upon traditionally human-delivered services. It's good to know about Dapr and restate.dev.

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62

Data Engineering Weekly

MARCH 23, 2025

I honestly don’t have a solid answer, but this blog is an excellent overview of upskilling. The author emphasizes the importance of mastering state management, understanding "local first" data processing (prioritizing single-node solutions before distributed systems), and leveraging an asset graph approach for data pipelines.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily.

Kafka 66

Kafka Datasets Metadata Utilities 66

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Many of our customers use multiple solutions—but want to consolidate data security, governance, lineage, and metadata management, so that they don’t have to work with multiple vendors.

Database 92

Database Cloud Systems Management 92

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. The commonly-accepted best practice in database system design for years is to use an exhaustive search strategy to consider all the possible variations of specific database operations in a query plan. Metadata Caching. More on this below.

Metadata 143

Metadata Coding SQL Database 143

phData: Data Engineering

NOVEMBER 1, 2024

This blog is a collection of those insights, but for the full trendbook, we recommend downloading the PDF. Strong data governance also lays the foundation for better model performance, cost efficiency, and improved data quality, which directly contributes to regulatory compliance and more secure AI systems. No problem!

Government 52

Government Data Governance Finance Metadata 52

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. This is crucial for applications that require up-to-date information, such as fraud detection systems or recommendation engines. What is Change Data Capture?

Kafka 50

Kafka MySQL Database Software Engineer 50

Picnic Engineering

FEBRUARY 6, 2025

In our previous blogs, we explored how Picnics Page Platform transformed the way we build new featuresenabling faster iteration, tighter collaboration and less feature-specific complexity. In this blog, well dive into how we configure the pages within Picnics store. Now, were bringing this same principle to our app.

Metadata 52

Metadata Architecture SQL Building 52

Tweag

MAY 16, 2023

That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure. A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.

MySQL 135

MySQL Metadata Data Validation Coding 135

Data Engineering Weekly

FEBRUARY 23, 2025

It covers nine categories: storage systems, data lake platforms, processing, integration, orchestration, infrastructure, ML/AI, metadata management, and analytics. I found the blog to be a comprehensive roadmap for data engineering in 2025. Let me know in the comments. link] All rights reserved ProtoGrowth Inc, India.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Ascend.io

DECEMBER 4, 2024

In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. On average, engineers spend over half of their time maintaining existing systems rather than developing new solutions. Instead of driving innovation, data engineers often find themselves bogged down with maintenance tasks.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Cloudera

MARCH 4, 2024

This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control. First, a set of initial metadata objects are created by the data steward.

Metadata 52

Metadata Data Governance Government Management 52

Data Engineering Podcast

FEBRUARY 26, 2023

What are the technical systems that you are relying on to power the different data domains? What is your philosophy on enforcing uniformity in technical systems vs. relying on interface definitions as the unit of consistency? What are the technical systems that you are relying on to power the different data domains?

Building 147

Building Machine Learning Metadata Data Integration 147

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 138

Utilities Systems Architecture Coding 138

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. All the above commands are very likely to be described in separate future blog posts, but right now let’s focus on the dataflow sample command.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Data Engineering Weekly

JUNE 16, 2024

link] Jack Vanlightly: A Cost Analysis Of Replication Vs. S3 Express One Zone In Transactional Data Systems S3 Express One Zone, with low latency and write ops certainty, is promising. Replication-based systems remain more economical at low to medium throughputs, especially with significant cross-AZ data transfer discounts.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

DoorDash Engineering

MARCH 14, 2023

When dealing with failures in a microservice system, localized mitigation mechanisms like load shedding and circuit breakers have always been used, but they may not be as effective as a more globalized approach. Each section is accompanied by real outages drawn from our past blog posts that can be explored in greater detail.

Java 135

Java Metadata Database Systems 135

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Late-arriving data is essentially delayed data due to system retries, network delays, batch processing schedules, system outages, delayed upstream workflows, or reconciliation in source systems.

Data Engineering 108

Data Engineering Data Engineer Engineering Metadata 108

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 113

Data Science Cloud Hadoop Metadata 113

Snowflake

JULY 25, 2024

Yet, while retrieval is a fundamental component of any AI application stack, creating a high-quality, high-performance RAG system remains challenging for most enterprises. It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata.

Unstructured Data 124

Unstructured Data Metadata Government SQL 124

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. An OS snapshot is a collection of bootfiles (initrd, vmlinuz), RPMs, and a few extra metadata.

Systems 55

Systems Metadata Datasets Designing 55

Ascend.io

DECEMBER 4, 2024

In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. On average, engineers spend over half of their time maintaining existing systems rather than developing new solutions. Instead of driving innovation, data engineers often find themselves bogged down with maintenance tasks.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Cloudera

AUGUST 10, 2021

Apache Ozone enhancements deliver full High Availability providing customers with enterprise-grade object storage and compatibility with Hadoop Compatible File System and S3 API. . We expand on this feature later in this blog. Figure 8: Data lineage based on Kafka Atlas Hook metadata. x, and 6.3.x, x, and 6.3.x,

Cloud 97

Cloud Kafka Metadata SQL 97

Maxime Beauchemin

JANUARY 20, 2017

This discipline also integrates specialization around the operation of so called “big data” distributed systems, along with concepts around the extended Hadoop ecosystem, stream processing, and in computation at scale. Those systems have been taught to normalize the data for storage on their own.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The author highlights Paimon’s consistency model by examining the metadata model.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Netflix Tech

SEPTEMBER 2, 2020

Edgar helps Netflix teams troubleshoot distributed systems efficiently with the help of a summarized presentation of request tracing, logs, analysis, and metadata. The more complex a system, the more places to look for clues. In an earlier blog post, we discussed Telltale , our health monitoring system.

Metadata 119

Metadata Transportation Engineering Systems 119

Cloudera

FEBRUARY 22, 2022

Companies such as Adobe , Expedia , LinkedIn , Tencent , and Netflix have published blogs about their Apache Iceberg adoption for processing their large scale analytics datasets. . In CDP we enable Iceberg tables side-by-side with the Hive table types, both of which are part of our SDX metadata and security framework.

Metadata 110

Metadata Datasets BI SQL 110