Data Engineering Weekly

JANUARY 12, 2025

Learn practical strategies to optimize Airflow performance and streamline operations: - Fine-tune configurations to enhance workflow efficiency - Automate Airflow deployments and manage users seamlessly - Monitor system health with advanced observability tools and alerts Join this live session and learn how to scale Airflow efficiently.

Pipeline-centric 57

Pipeline-centric Data Engineer Data Engineering Engineering 57

Snowflake

NOVEMBER 2, 2023

To give customers flexibility for how they fit Snowflake into their architecture, Iceberg Tables can be configured to use either Snowflake or an external service like AWS Glue as the tables’s catalog to track metadata, with an easy one-line SQL command to convert to Snowflake in a metadata-only operation.

Data Lake 114

Data Lake Data Warehouse Cloud Unstructured Data 114

AltexSoft

JULY 27, 2021

You’ll learn about the types of recommender systems, their differences, strengths, weaknesses, and real-life examples. Personalization and recommender systems in a nutshell. Primarily developed to help users deal with a large range of choices they encounter, recommender systems come into play. Amazon, Booking.com) and.

Machine Learning 76

Machine Learning Systems Algorithm Deep Learning 76

Engineering at Meta

MARCH 18, 2024

Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. We designed the system to support service-level guarantees on log freshness, completeness, durability, query latency, and query result completeness. Each log line can have zero or more metadata key-value pairs attached to it.

Engineering 94

Engineering Metadata Architecture Designing 94

Data Engineering Podcast

JULY 1, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. When doing data collection from various sources, how do you ensure that intellectual property rights are respected?

Metadata 100

Metadata Machine Learning Data Preparation Data Collection 100

Cloudera

JUNE 7, 2022

We live in a hybrid data world. In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB.

IT 112

IT Unstructured Data Data Architecture Government 112

AltexSoft

JUNE 7, 2021

You don’t need to archive or clean data before loading. The system automatically replicates information to prevent data loss in the case of a node failure. To understand how the entire mechanism works, we need to get familiar with Hadoop structure and key parts. A file stored in the system ?an’t fail-safety.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Engineering at Meta

FEBRUARY 6, 2024

DotSlash handles transparently fetching, decompressing, and verifying the appropriate remote artifact for the current operating system and CPU. Our continuous integration (CI) system supports special configuration for DotSlash jobs where a user must specify: A set of builds to run (these can span multiple platforms).

Metadata 120

Metadata Coding Building Project 120

Data Engineering Podcast

FEBRUARY 3, 2019

Open Context is an open access data publishing service for archaeology. It started because we need better ways of dissminating structured data and digital media than is possible with conventional articles, books and reports. What are your protocols for determining which data sets you will work with?

Digital Media 100

Digital Media Media PostgreSQL Datasets 100

Netflix Tech

DECEMBER 14, 2018

In the previous blog posts in this series, we introduced the N etflix M edia D ata B ase ( NMDB ) and its salient “Media Document” data model. In this post we will provide details of the NMDB system architecture beginning with the system requirements?—?these key value stores generally allow storing any data under a key).

Media 97

Media Database Metadata Data Schemas 97

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Netflix Tech

MARCH 5, 2019

This operational component places some cognitive load on our engineers, requiring them to develop deep understanding of telemetry and alerting systems, capacity provisioning process, security and reliability best practices, and a vast amount of informal knowledge about the cloud infrastructure.

Cloud 74

Cloud Building Amazon Web Services Metadata 74

Snowflake

MARCH 30, 2023

For this reason, a new data management for ML framework has emerged to help manage this complexity: the “feature store.” Feature store As described in Tecton’s blog , a feature store is a data management system for managing ML feature pipelines, including the management of feature engineering code and data.

Engineering 104

Engineering Raw Data Data Science Machine Learning 104

Cloudera

SEPTEMBER 17, 2020

By enabling their event analysts to monitor and analyze events in real time, as well as directly in their data visualization tool, and also rate and give feedback to the system interactively, they increased their data to insight productivity by a factor of 10. .

Data Warehouse 81

Data Warehouse Unstructured Data Pharmaceutical MySQL 81

Cloudera

NOVEMBER 15, 2024

Cyber defenders struggle with: Too much data: Cybersecurity tools generate an overwhelming volume of log data, including Domain Name Service (DNS) records, firewall logs, and more. All of this data is essential for investigations and threat hunting, but existing systems often struggle to manage it efficiently.

Metadata 59

Metadata Unstructured Data Data Lake Government 59

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. What is a Data Lake? What are Data Modeling Methodologies, and Why Are They Important for a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Netflix Tech

OCTOBER 27, 2020

The data warehouse is not designed to serve point requests from microservices with low latency. Therefore, we must efficiently move data from the data warehouse to a global, low-latency and highly-reliable key-value store. truncate or drop a table) this allows us to cheaply recycle old versions of the data.

Data Warehouse 100

Data Warehouse Datasets Data Big Data 100

AltexSoft

NOVEMBER 10, 2021

In a nutshell, the lakehouse system leverages low-cost storage to keep large volumes of data in its raw formats just like data lakes. At the same time, it brings structure to data and empowers data management features similar to those in data warehouses by implementing the metadata layer on top of the store.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

AltexSoft

AUGUST 22, 2022

A data fabric is an architecture design presented as an integration and orchestration layer built on top of multiple disjointed data sources like relational databases , data warehouses , data lakes, data marts , IoT , legacy systems, etc., to provide a unified view of all enterprise data.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

Netflix Tech

OCTOBER 31, 2018

the Media Timeline Data Model In the previous post in this series, we described some important Netflix business needs as well as traits of the media data system?—?called The curious reader might have noticed that a majority of these characteristics relate to properties of the data managed by NMDB.

Media 54

Media Metadata Data MongoDB 54

Ascend.io

FEBRUARY 28, 2024

And crucially, what does the future hold for data engineering in an AI-driven world? While data engineering and Artificial Intelligence (AI) may seem like distinct fields at first glance, their symbiosis is undeniable. The foundation of any AI system is high-quality data.

Data Engineer 52

Data Engineer Data Engineering Engineering Metadata 52

Zalando Engineering

JANUARY 30, 2023

Our Incident Playbooks cover emergency procedures to initiate in case a certain set of conditions is met, for example when one of our systems is overloaded and the existing resiliency measures (e.g. When the bigger system context is considered, there are more options available to mitigate issues. processing of price updates).

Management 56

Management Metadata Software Engineer Software Engineering 56

Towards Data Science

FEBRUARY 15, 2023

The Data Lake architecture was proposed in a period of great growth in the data volume, especially in non-structured and semi-structured data, when traditional Data Warehouse systems start to become incapable of dealing with this demand. The data became useless. delta_table.history().select("version",

Data Lake 98

Data Lake Data Warehouse Hadoop Architecture 98

Towards Data Science

MARCH 5, 2024

The storage system is using Capacitor, a proprietary columnar storage format by Google for semi-structured data and the file system underneath is Colossus, the distributed file system by Google. depending on location) BigQuery maintains a lot of valuable metadata about tables, columns and partitions.

Bytes 97

Bytes Google Cloud Cloud Storage Utilities 97

ProjectPro

FEBRUARY 16, 2023

Data integration with ETL has evolved from structured data stores with high computing costs to natural state storage with read operation alterations thanks to the agility of the cloud. Data integration with ETL has changed in the last three decades. AWS Glue has a central metadata repository called the Glue catalog.

AWS 52

AWS Data Management ETL Tools Management 52

Sync Computing

NOVEMBER 7, 2024

Despite these limitations, data warehouses, introduced in the late 1980s based on ideas developed even earlier, remain in widespread use today for certain business intelligence and data analysis applications. While data warehouses are still in use, they are limited in use-cases as they only support structured data.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

Wayne Yaddow

MARCH 14, 2025

Snapshot testing augments debugging capabilities by recording past table states, facilitating the identification of unforeseen spikes, declines, or abnormalities before their effect on production systems. Data freshness propagation: No automatic tracking of data propagation delays across multiplemodels.

Unstructured Data 52

Unstructured Data SQL Data Pipeline Data Validation 52

Data Engineering Weekly

MARCH 24, 2023

Data creation is often the differentiator between the success & the failure of a data team. A business process or workflow engine is a software system that enables businesses to execute well-defined steps to complete a user’s intention. Event Sourcing Change Data Capture [CDC] Outbox pattern 1.

Engineering 67

Engineering Data Transportation Database 67

AltexSoft

AUGUST 29, 2023

Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Watch our video explaining how data engineering works.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

ProjectPro

AUGUST 11, 2021

Data Lake vs Data Warehouse - The Differences Before we closely analyse some of the key differences between a data lake and a data warehouse, it is important to have an in depth understanding of what a data warehouse and data lake is. Data Lake vs Data Warehouse - The Introduction What is a Data warehouse?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Rockset

DECEMBER 17, 2021

Here’s the final architecture: I’ve been doing some flavour of systems integration for the past 15 years, and usually I finish a project and think “it shouldn’t have taken that much effort”. For those unfamiliar, DynamoDB makes database scalability a breeze, but with some major caveats.

SQL 52

SQL NoSQL Database Design Metadata 52

Towards Data Science

JANUARY 17, 2024

Parquet vs ORC vs Avro vs Delta Lake Photo by Viktor Talashuk on Unsplash The big data world is full of various storage systems, heavily influenced by different file formats. These are key in nearly all data pipelines, allowing for efficient data storage and easier querying and information extraction.

Big Data 97

Big Data Data Data Storage SQL 97

AltexSoft

JULY 29, 2022

A Hadoop cluster is a group of computers called nodes that act as a single centralized system working on the same task. a client or edge node serves as a gateway between a Hadoop cluster and outer systems and applications. It loads data and grabs the results of the processing staying outside the master-slave hierarchy.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

JANUARY 31, 2023

Key features Hadoop RDBMS Overview Hadoop is an open-source software collection that links several computers to solve problems requiring large quantities of data and processing. RDBMS is a part of system software used to create and manage databases based on the relational model. RDBMS stores structured data.

Big Data 52

Big Data Hadoop Relational Database AWS 52

ProjectPro

OCTOBER 15, 2014

Hive- Performance Benchmarking Hive vs Pig Pig vs Hive - Differences Pig Hive Procedural Data Flow Language Declarative SQLish Language For Programming For creating reports Mainly used by Researchers and Programmers Mainly used by Data Analysts Operates on the client side of a cluster. Does not have a dedicated metadata database.

Hadoop 52

Hadoop Java Unstructured Data SQL 52

AltexSoft

SEPTEMBER 23, 2021

The larger the company, the more data it has to generate actionable insights. Because it is scattered across disparate systems, hardly available for analytical apps. Evidently, common storage solutions fail to provide a unified data view and meet the needs of companies for seamless data flow. Data lake vs data hub.

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52

Silectis

JANUARY 3, 2021

” Artificial Intelligence AI is a broad term used to describe engineered systems that have been taught to do a task that typically requires human intelligence. BI (Business Intelligence) Strategies and systems used by enterprises to conduct data analysis and make pertinent business decisions.

Data Engineer 40

Data Engineer Data Engineering Engineering Hadoop 40