Data Engineering Weekly

DECEMBER 28, 2024

Investment in an Agent Management System (AMS) is crucial, as it offers a framework for scaling, monitoring, and refining AI agents. AI engineers, in particular, will find their skills in high demand as they navigate managing and optimizing agents to ensure reliability within enterprise systems.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

Netflix Tech

MARCH 28, 2025

By Ko-Jen Hsiao , Yesu Feng and Sudarshan Lamkhede Motivation Netflixs personalized recommender system is a complex system, boasting a variety of specialized machine learned models each catering to distinct needs including Continue Watching and Todays Top Picks for You. Refer to our recent overview for more details).

Metadata 66

Metadata Bytes Data Mining Entertainment 66

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode Ian Schweer shares his experiences at Riot Games supporting player-focused features such as machine learning models and recommeder systems that are deployed as part of the game binary. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Precisely

OCTOBER 31, 2024

With data volumes skyrocketing, and complexities increasing in variety and platforms, traditional centralized data management systems often struggle to keep up. A data fabric weaves together different data management tools, metadata, and automation to create a seamless architecture.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

Engineering at Meta

JANUARY 22, 2025

It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems. It enhances the traceability of data flows within systems, ultimately empowering developers to swiftly implement privacy controls and create innovative products. Hack, C++, Python, etc.)

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Netflix Tech

JANUARY 6, 2025

In this case, the main stakeholders are: - Title Launch Operators Role: Responsible for setting up the title and its metadata into our systems. In this context, were focused on developing systems that ensure successful title launches, build trust between content creators and our brand, and reduce engineering operational overhead.

Metadata 87

Metadata Algorithm Systems Building 87

Knowledge Hut

FEBRUARY 1, 2024

Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems? Recommender systems have two main categories: content-based & collaborative filtering.

Systems 98

Systems Entertainment Algorithm Datasets 98

Engineering at Meta

JANUARY 21, 2025

Because there are so many different things happening in these systems powered by so many different technologies. Strobelight also has concurrency rules and a profiler queuing system. This provides just the right amount of data without impacting the profiled services or overburdening the systems that store Strobelight data.

Technology 114

Technology Metadata Utilities Engineering 114

ThoughtSpot

NOVEMBER 5, 2024

ThoughtSpot prioritizes the high availability and minimal downtime of our systems to ensure a seamless user experience. In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. What is Atlas?

Metadata 52

Metadata PostgreSQL Java Database 52

Cloudera

JANUARY 26, 2024

This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control. First, a set of initial metadata objects are created by the data steward.

Metadata 73

Metadata Data Governance Government Management 73

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Does it sound familiar?

Systems 98

Systems Raw Data Metadata Data Cleanse 98

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Systems 100

Systems Metadata Building MongoDB 100

Snowflake

MARCH 6, 2025

Meanwhile, operations teams use entity extraction on documents to automate workflows and enable metadata-driven analytical filtering. As data volumes grow and AI automation expands, cost efficiency in processing with LLMs depends on both system architecture and model flexibility.

Unstructured Data 97

Unstructured Data Medical Media Data Workflow 97

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata BI Data Lake Business Intelligence 84

Snowflake

DECEMBER 4, 2024

Beyond working with well-structured data in a data warehouse, modern AI systems can use deep learning and natural language processing to work effectively with unstructured and semi-structured data in data lakes and lakehouses.

Unstructured Data 97

Unstructured Data Data Lake Deep Learning Structured Data 97

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems. ETL workflows), as well as downstream (e.g.

Systems 94

Systems Media Machine Learning Data Warehouse 94

Data Engineering Weekly

MARCH 5, 2025

Hadoop achieved this through distributed processing and storage, using a framework called MapReduce and the Hadoop Distributed File System (HDFS). This ecosystem includes: Catalogs: Services that manage metadata about Iceberg tables (e.g., If not handled correctly, managing this metadata can become a bottleneck.

Hadoop 57

Hadoop Metadata Data Ingestion Data Governance 57

Data Engineering Podcast

APRIL 24, 2022

WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model. You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms.

Machine Learning 100

Machine Learning Systems Data Lake Java 100

Data Engineering Podcast

AUGUST 13, 2022

The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Sifflet also offers a 2-week free trial. Sifflet also offers a 2-week free trial.

Metadata 100

Metadata MongoDB MySQL Scala 100

Data Engineering Weekly

NOVEMBER 24, 2024

The system leverages a combination of an event-based storage model in its TimeSeries Abstraction and continuous background aggregation to calculate counts across millions of counters efficiently. Grab has enhanced its LLM-powered data classification system, Metasense, to improve accuracy and minimize manual workload.

Data Engineering 57

Data Engineering Data Engineer Engineering Insurance 57

Pinterest Engineering

APRIL 4, 2025

To represent each Pin, we use the following varied set of text features derived from metadata, the image itself, as well as user-curated data. A diagram of the search relevance system at Pinterest is shown in Figure 3. Figure 3: Diagram of the proposed search relevance system at Pinterest.

Machine Learning 54

Machine Learning Metadata Architecture Datasets 54

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. We also considered caching data logs in an online system capable of supporting a range of indexed per-user queries. What are data logs?

Accessible 80

Accessible Accessibility Raw Data Data Warehouse 80

Precisely

NOVEMBER 18, 2024

With the surge of new tools, platforms, and data types, managing these systems effectively is an ongoing challenge. Focus on metadata management. As Yoğurtçu points out, “metadata is critical” for driving insights in AI and advanced analytics. And context also enhances the large language models.

Data Analytics 72

Data Analytics Data Governance Data Integration Government 72

Data Engineering Podcast

JUNE 16, 2024

What are the other systems that feed into and rely on the Trino/Iceberg service? what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? Email hosts@dataengineeringpodcast.com with your story.

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Many of our customers use multiple solutions—but want to consolidate data security, governance, lineage, and metadata management, so that they don’t have to work with multiple vendors.

Database 92

Database Cloud Systems Management 92

Start Data Engineering

NOVEMBER 21, 2024

A compute engine is a system that transforms data 3.1.2. Metadata catalog stores information about datasets 3.1.3. Analytical databases aggregate large amounts of data 3. Most platforms enable you to do the same thing but have different strengths 3.1. Understand how the platforms process data 3.1.1.

Metadata 130

Metadata Datasets SQL Database 130

Precisely

FEBRUARY 26, 2025

Automated metadata management – AI-generated catalog asset descriptions significantly reduce manual efforts and improve metadata quality – enabling teams to focus on more strategic tasks. With the ability to turn functionality on or off based on business requirements, you gain full control over when and how AI is applied.

Data Integration 64

Data Integration Data Management Management Data Governance 64

phData: Data Engineering

NOVEMBER 1, 2024

Strong data governance also lays the foundation for better model performance, cost efficiency, and improved data quality, which directly contributes to regulatory compliance and more secure AI systems. VP of Architecture, Healthcare Industry Organizations will focus more on metadata tagging of existing and new content in the coming years.

Government 52

Government Data Governance Finance Metadata 52

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Atlan is the metadata hub for your data ecosystem. What are some of the unique characteristics of that information?

Building 130

Building IT Metadata MongoDB 130

Data Engineering Weekly

FEBRUARY 18, 2025

Kafka is designed to be a black box to collect all kinds of data, so Kafka doesn't have built-in schema and schema enforcement; this is the biggest problem when integrating with schematized systems like Lakehouse. If you want to build OLAP systems for low-latency complex queries, use Pinot. When to use Fluss vs Apache Pinot?

Kafka 73

Kafka Lambda Architecture SQL Architecture 73

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Cloudera

NOVEMBER 13, 2020

The commonly-accepted best practice in database system design for years is to use an exhaustive search strategy to consider all the possible variations of specific database operations in a query plan. Metadata Caching. See the performance results below for an example of how metadata caching helps reduce latency.

Metadata 143

Metadata Coding SQL Database 143

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily.

Kafka 66

Kafka Datasets Metadata Utilities 66

Data Engineering Weekly

APRIL 6, 2025

The article summarizes the recent macro trends in AI and data engineering, focusing on Vibe coding, human-in-the-loop system design, and rapid simplification of developer tooling. Kudos to the Grab team for building a docs-as-code system. The Grab blog delights me since I have tried to do this many times.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Ascend.io

DECEMBER 4, 2024

On average, engineers spend over half of their time maintaining existing systems rather than developing new solutions. Create a Plan for Integration: Automation tools need to work seamlessly with existing systems to be effective. Instead of driving innovation, data engineers often find themselves bogged down with maintenance tasks.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Snowflake

MAY 15, 2024

And that gap is directly affecting the performance and reliability of AI systems everywhere. By continuously monitoring metrics, metadata, lineage, and logs from across your data infrastructure and using ML-based anomaly detection to detect issues, they help data teams know about and resolve issues quickly.

Cloud 126

Cloud Metadata Data Pipeline Government 126