Data Engineering Podcast

AUGUST 24, 2020

The key to those solutions is a robust and flexible metadata management system. LinkedIn has gone through several iterations on the most maintainable and scalable approach to metadata, leading them to their current work on DataHub. What were you using at LinkedIn for metadata management prior to the introduction of DataHub?

Metadata 100

Metadata Management Kafka Data Engineer 100

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. How is everyone going to find the data they need, and understand it?

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Cloudera

SEPTEMBER 15, 2022

In this blog post, we will talk about a single Ozone cluster with the capabilities of both Hadoop Core File System (HCFS) and Object Store (like Amazon S3). A unified storage architecture that can store both files and objects and provide a flexible, scalable, and high-performance system. Bucket types. release version.

Systems 87

Systems Hadoop Metadata Telecommunication 87

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. An OS snapshot is a collection of bootfiles (initrd, vmlinuz), RPMs, and a few extra metadata.

Systems 55

Systems Metadata Datasets Designing 55

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 83

Metadata BI Data Lake Business Intelligence 83

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. So which static site generator to choose?

Engineering 40

Engineering Bytes AWS Python 40

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Many of our customers use multiple solutions—but want to consolidate data security, governance, lineage, and metadata management, so that they don’t have to work with multiple vendors.

Database 91

Database Cloud Systems Management 91

Cloudera

NOVEMBER 15, 2024

All of this data is essential for investigations and threat hunting, but existing systems often struggle to manage it efficiently. Here’s how Cloudera makes it possible: One unified system: Cloudera’s open data lakehouse consolidates all critical log data into one system.

Metadata 77

Metadata Unstructured Data Data Lake Government 77

Pinterest Engineering

JANUARY 10, 2023

Liang Ma | Software Engineer, Core Eng; Wei Zhu | Software Engineer, Observability In early 2020, during a critical iOS out of memory incident (we have a blogpost for that), we realized that we didn’t have much visibility of how the app is running or a good system to look up for monitoring and troubleshooting. Nothing else is required.

Systems 57

Systems Building Software Engineering Software Engineer 57

Data Engineering Weekly

JUNE 16, 2024

link] Jack Vanlightly: A Cost Analysis Of Replication Vs. S3 Express One Zone In Transactional Data Systems S3 Express One Zone, with low latency and write ops certainty, is promising. Replication-based systems remain more economical at low to medium throughputs, especially with significant cross-AZ data transfer discounts.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Cloudera

DECEMBER 7, 2020

Apache Hadoop Distributed File System (HDFS) is the most popular file system in the big data world. The Apache Hadoop File System interface has provided integration to many other popular storage systems like Apache Ozone, S3, Azure Data Lake Storage etc. Migrating file systems thus requires a metadata update. .

Systems 60

Systems Hadoop Metadata Datasets 60

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The author highlights Paimon’s consistency model by examining the metadata model.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 138

Algorithm Metadata Kafka Systems 138

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. This architecture shift greatly reduced the processing latency and increased system resiliency.

Process 93

Process Pipeline-centric Media Metadata 93

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Late-arriving data is essentially delayed data due to system retries, network delays, batch processing schedules, system outages, delayed upstream workflows, or reconciliation in source systems.

Data Engineering 107

Data Engineering Data Engineer Engineering Metadata 107

Pinterest Engineering

OCTOBER 2, 2024

Catalog/Metadata Management , which is responsible for storing and serving table metadata, including ownership, schemas, SLAs, etc., and also providing runtime metadata about the platform Security and Auditability , which ensures GDPR compliance, advanced levels of auditability, security, AAA, etc.

Data Management 56

Data Management Management Metadata SQL 56

Netflix Tech

SEPTEMBER 18, 2024

message Item ( Bytes key, Bytes value, Metadata metadata, Integer chunk ) Database Agnostic Abstraction The KV abstraction is designed to hide the implementation details of the underlying database, offering a consistent interface to application developers regardless of the optimal storage system for that use case.

Bytes 99

Bytes Metadata Database Data 99

Snowflake

JULY 25, 2024

Yet, while retrieval is a fundamental component of any AI application stack, creating a high-quality, high-performance RAG system remains challenging for most enterprises. It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata.

Unstructured Data 111

Unstructured Data Metadata Government SQL 111

Data Engineering Weekly

SEPTEMBER 15, 2024

link] Miro: Data Products Reliability - The Power of Metadata Miro writes about its adoption of Data Contracts. impactdatasummit.com Ravi Vedula: The evolution of a data platform - The story of the Microsoft IDEAS team Microsoft IDEAS (Insights, Data, Engineering, Analytics, Systems) team writes about the evolution of the data platform.

Data Engineering 67

Data Engineering Data Engineer Engineering Data Lake 67

Tweag

MAY 16, 2023

That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure. A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.

MySQL 135

MySQL Metadata Data Validation Coding 135

Data Engineering Podcast

FEBRUARY 26, 2023

What are the technical systems that you are relying on to power the different data domains? What is your philosophy on enforcing uniformity in technical systems vs. relying on interface definitions as the unit of consistency? What are the technical systems that you are relying on to power the different data domains?

Building 147

Building Machine Learning Metadata Data Integration 147

Netflix Tech

OCTOBER 8, 2024

In previous blog posts, we introduced the Key-Value Data Abstraction Layer and the Data Gateway Platform , both of which are integral to Netflix’s data architecture. Note: Contrary to what the name may suggest, this system is not built as a general-purpose time series database.

Bytes 94

Bytes Datasets Metadata Data 94

Data Engineering Weekly

AUGUST 25, 2024

Google’s search ranking system is a complex, multi-step process that begins with indexing new content, assigning it a unique DocID, and calculating its relevance based on keyword presence. The blog is a good overview of various components in a typical data stack. This is a must-read for this week.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

Databand.ai

AUGUST 30, 2023

The importance of data quality cannot be overstated, as poor-quality data can result in incorrect conclusions, inefficient operations, and a lack of trust in the information provided by a company’s systems. Consistency: The uniformity of data across different sources or systems.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Data Engineering Weekly

NOVEMBER 3, 2024

Foundation Capital: A System of Agents brings Service-as-Software to life software is no longer simply a tool for organizing work; software becomes the worker itself, capable of understanding, executing, and improving upon traditionally human-delivered services. It's good to know about Dapr and restate.dev.

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. The data files and metadata files in Iceberg format are immutable.

Java 70

Java Metadata PostgreSQL Data Warehouse 70

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Rockset

AUGUST 27, 2023

This blog attempts to arm you with some knowledge of your future, the problems you will face, and questions you may not know yet that you need to ask. Vector search ≠ vector database Vector search and all the associated clever algorithms are the central intelligence of any system trying to leverage vectors.

Metadata 52

Metadata Database Algorithm SQL 52

Pinterest Engineering

OCTOBER 11, 2024

In Part 2 of our blog series, we described how we were able to integrate Ray(™) into our existing ML infrastructure. In this blog post, we will discuss a second type of popular application of Ray(™) at Pinterest: offline batch inference of ML models. Results are from vLLM team’s blog.

Datasets 63

Datasets Software Engineering Software Engineer Metadata 63

Data Engineering Weekly

OCTOBER 6, 2024

link] Medium: Learnings from optimising 22 of our most expensive Snowflake pipelines A blog about real-world learning is always a delight to read. The author highlights the key benefits of thinking and designing systems to support quality and compliance upfront (shift-left) rather than later. Sonnet, GPT-4o, Mistral Large 2, Llama 3.1

Data Engineering 62

Data Engineering Data Engineer Engineering Data Warehouse 62

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 99

Metadata Data Pipeline Scala Data Workflow 99

Christophe Blefari

JULY 13, 2024

AI News 🤖 OpenAI — Always the biggest news provider, announcements or dramas OpenAI was hacked, revealing internal secrets and raising national security concerns — The hacker reached OpenAI’s internal messaging systems early last year, stealing details of how OpenAI's technologies work from employees.

Kafka 130

Kafka AWS Data Database 130

Cloudera

JULY 24, 2024

Citizens of large cities heavily depend on their local metro systems to plan their day-to-day lives. Managing IT infrastructure starts to look like a city’s transportation infrastructure when you realize that end users also depend on the reliability of IT systems to complete work and get home on time.

Transportation 78

Transportation Metadata Cloud Systems 78

Knowledge Hut

APRIL 25, 2024

Kubernetes (sometimes shortened to K8s with the 8 standing for the number of letters between the “K” and the “s”) is an open-source system to deploy, scale, and manage containerized applications anywhere. apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 137

Utilities Systems Architecture Coding 137

Data Engineering Weekly

NOVEMBER 11, 2022

Not too long ago, I worked for a famous messaging platform; You may be typing on it as you read this blog. I took a break from data engineering and focused on system monitoring and observability for some time. Little did I know I would be building data pipelines to measure system efficiency instead of measuring business efficiency.

Systems 40

Systems High Quality Data Consulting Metadata 40

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? It bridges gaps in data ecosystems, ensuring consistency and synchronisation across systems. Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80