Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The CSP engine is powered by Apache Flink, which is the best-in-class processing engine for stateful streaming pipelines. Currently, Iceberg support in CSP is in technical preview mode.

Process 113

Process SQL Kafka Database 113

Netflix Tech

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Process 86

Process Data Pipeline Datasets SQL 86

phData: Data Engineering

SEPTEMBER 19, 2023

However, to fully harness the potential of a data lake, effective data modeling methodologies and processes are crucial. With many data modeling methodologies and processes available, choosing the right approach can be daunting. Efficiency through being able to streamline data storage and retrieval processes.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

LinkedIn Engineering

JUNE 15, 2023

To remove this bottleneck, we built AvroTensorDataset , a TensorFlow dataset for reading, parsing, and processing Avro data. If greater than one, records in files are processed in parallel. The bytes are decoded based on the provided features metadata (i.e. Zero shuffle buffer size means shuffle is disabled.

Datasets 102

Datasets Bytes Process Data Ingestion 102

LinkedIn Engineering

JANUARY 19, 2024

This multi-entity handover process involves huge amounts of data updating and cloning. Data consistency, feature reliability, processing scalability, and end-to-end observability are key drivers to ensuring business as usual (zero disruptions) and a cohesive customer experience. Push for eventual success of the request.

Recruitment 88

Recruitment Data Process Process Kafka 88

Jesse Anderson

NOVEMBER 14, 2023

A Guest Post by Ole Olesen-Bagneux In this blog post I would like to describe a new data team, that I call ‘the data discovery team’. And that is what the data discovery team that I propose in this blog post should work on: searching for data. In an enterprise data reality, searching for data is a bit of a hassle.

Metadata 147

Metadata Data Science Big Data Data 147

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 83

Metadata BI Data Lake Business Intelligence 83

Snowflake

AUGUST 7, 2024

As we describe in this blog post , the top-k feature uses runtime information — namely, the current contents of the top-k elements — to skip micro-partitions where we can guarantee that they won’t contribute to the overall result. Snowflake starts processing those partitions first. on average, with some queries also reaching up to 99.8%

Metadata 61

Metadata Algorithm Utilities Process 61

LinkedIn Engineering

MARCH 23, 2023

Co-Authors: Yuhong Cheng , Shangjin Zhang , Xinyu Liu, and Yi Pan Efficient data processing is crucial in reducing learning curves, simplifying maintenance efforts, and decreasing operational complexity. By unifying these pipelines, we have saved 94% of processing time. Samza , Spark and Apache Flink ).

Process 97

Process Lambda Architecture Kafka Datasets 97

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. It's actively developed.

Engineering 40

Engineering Bytes AWS Python 40

Cloudera

NOVEMBER 15, 2024

The manual process of switching between tools slows down their work, often leaving them reliant on rudimentary methods of keeping track of their findings. The metadata-driven approach ensures quick query planning so defenders don’t have to deal with slow processes when they need fast answers.

Metadata 77

Metadata Unstructured Data Data Lake Government 77

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Let’s dive in! What is late-arriving data? How does late-arriving data impact us?

Data Engineering 107

Data Engineering Data Engineer Engineering Metadata 107

Data Engineering Weekly

JUNE 24, 2024

The blog contains a summary of each talk and a link to the YouTube channel with all the talks. The blog details the classification model, training approach and historical data analysis. Question to the readers, what do you think of the current state of real-time data processing engines? Are there enough usecases?

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Data Engineering Weekly

JUNE 16, 2024

Get Your Copy → Walmart: Reliably Processing Trillions of Kafka Messages Per Day Consumer rebalancing and Head-of-line (HOL) blocking are some of the most common challenges while operating Kafka at scale. The blog explains KIP-932 and its potential benefits. link] Picnic: Open-sourcing dbt-score: lint model metadata with ease!

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Cloudera

JULY 10, 2024

Observability for your most secure data For your most sensitive, protected data, we understand even the metadata and telemetry about your workloads must be kept under close watch, and it must stay within your secured environment. The post Introducing Cloudera Observability Premium appeared first on Cloudera Blog.

Metadata 98

Metadata Cloud Management IT 98

Pinterest Engineering

OCTOBER 11, 2024

In Part 2 of our blog series, we described how we were able to integrate Ray(™) into our existing ML infrastructure. In this blog post, we will discuss a second type of popular application of Ray(™) at Pinterest: offline batch inference of ML models. These offline batch inference jobs can be both I/O and compute intensive.

Datasets 63

Datasets Software Engineering Software Engineer Metadata 63

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing.

Metadata 99

Metadata Data Pipeline Scala Data Workflow 99

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The author highlights Paimon’s consistency model by examining the metadata model.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Maxime Beauchemin

JANUARY 7, 2018

Batch data processing  — historically known as ETL —  is extremely challenging. In this post, we’ll explore how applying the functional programming paradigm to data engineering can bring a lot of clarity to the process. The greater the claim made using analytics, the greater the scrutiny on the process should be.

Data Engineering 183

Data Engineering Data Engineer Data Process Process 183

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 99

Building Metadata Cloud Storage AWS 99

Snowflake

AUGUST 7, 2024

As we describe in this blog post , the top-k feature uses runtime information — namely, the current contents of the top-k elements — to skip micro-partitions where we can guarantee that they won’t contribute to the overall result. Snowflake starts processing those partitions first. on average, with some queries also reaching up to 99.8%

Metadata 52

Metadata Algorithm Utilities Architecture 52

Rockset

MARCH 18, 2024

Flink is one of the most popular stream processing technologies, ranked as a top five Apache project and backed by a diverse committer community including Alibaba and Apple. It powers steam processing at many companies including Uber, Netflix, and Linkedin.

Cloud 64

Cloud Building Metadata Kafka 64

Snowflake

JULY 25, 2024

It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata. The service automatically indexes and embeds your data in an incremental fashion, meaning it only processes changed rows from the underlying data source.

Unstructured Data 111

Unstructured Data Metadata Government SQL 111

phData: Data Engineering

NOVEMBER 8, 2024

Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms. In this blog, we will discuss: What is the Open Table format (OTF)? Though basic and easy to use, traditional table storage formats struggle to keep up.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Pinterest Engineering

OCTOBER 2, 2024

Catalog/Metadata Management , which is responsible for storing and serving table metadata, including ownership, schemas, SLAs, etc., and also providing runtime metadata about the platform Security and Auditability , which ensures GDPR compliance, advanced levels of auditability, security, AAA, etc. SQL is not very easy to parse.

Data Management 56

Data Management Management Metadata SQL 56

Netflix Tech

SEPTEMBER 18, 2024

message Item ( Bytes key, Bytes value, Metadata metadata, Integer chunk ) Database Agnostic Abstraction The KV abstraction is designed to hide the implementation details of the underlying database, offering a consistent interface to application developers regardless of the optimal storage system for that use case. . "persistence_configuration":[

Bytes 99

Bytes Metadata Database Data 99

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 96

Management Kafka Metadata Media 96

Databand.ai

AUGUST 30, 2023

It utilizes various techniques to identify and resolve data quality issues, ensuring that high-quality data is used for business processes and decision-making. A shorter time-to-value indicates that your organization is efficient at processing and analyzing data for decision-making purposes.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Tweag

APRIL 19, 2023

To allow innovation in medical imaging with AI, we need efficient and affordable ways to store and process these WSIs at scale. A single hospital makes many captures a day, producing terabytes of such data to store and process. Much like Spark, its module ray.data is focused on loading and processing data as scale. width , spec.

Medical 60

Medical Process Cloud Bytes 60

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Create Iceberg materialized view For the examples in this blog, we will use three tables from the TPC-DS dataset as our base tables: store_sales, customer and date_dim. Both full and incremental rebuild of the materialized view are supported.

Metadata 90

Metadata Data Warehouse BI AWS 90

Cloudera

OCTOBER 10, 2024

Compaction is a process that rewrites small files into larger ones to improve performance. Users may want to perform table maintenance functions, like expiring snapshots, removing old metadata files, and deleting orphan files, to optimize storage utilization and improve performance.

IT 84

IT Data Lake Metadata Data Warehouse 84

Cloudera

JULY 19, 2023

It is a replicated, highly-available service that is responsible for managing the metadata for all objects stored in Ozone. In this blog post, we will highlight the work done recently to improve the performance of Ozone Manager to scale to exabytes of data. The hardware specifications are included at the end of this blog.

Management 70

Management Metadata Datasets Architecture 70

Christophe Blefari

JULY 13, 2024

This process involves converting files, regardless of their format, into text. It provides abstractions and tools for the translation of lakehouse table format metadata. You can also read Timo detailed post on Mixpanel blog and why SCD are the best thing for product analytics. His chart are great.

Kafka 130

Kafka AWS Data Database 130

Cloudera

JULY 24, 2024

Cloudera helps you with this maintenance by delivering improvements and vulnerability patches in Service Packs and Cumulative Hotfixes (CHFs) < Cumulative Hotfix Blog > Although applying Service Packs and CHFs is a straightforward process, you do need to restart services. To learn more, visit our product page.

Transportation 78

Transportation Metadata Cloud Systems 78

Data Engineering Podcast

FEBRUARY 26, 2023

In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When is a data mesh the wrong choice?

Building 147

Building Machine Learning Metadata Data Integration 147

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? CDC also plays a crucial role in data integration and ETL processes. It enables automation in business processes, event-driven applications, and microservices, reacting to data changes in near real-time. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Weekly

NOVEMBER 3, 2024

The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. link] Gunnar Morling: Revisiting the Outbox Pattern The blog is an excellent summary of the path we crossed with the outbox pattern and the challenges ahead.

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62