Data Engineering Podcast

SEPTEMBER 14, 2020

Tree Schema is a data catalog that is making metadata management accessible to everyone. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home.

Process 100

Process Unstructured Data Metadata Data Engineering 100

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Let’s dive in! What is late-arriving data? How does late-arriving data impact us?

Data Engineering 109

Data Engineering Data Engineer Engineering Metadata 109

Netflix Tech

JUNE 1, 2023

New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. Next, we launched a Mantis job that processed all requests in the stream and replayed them in a duplicate production environment created for replay traffic. We used Elasticsearch dashboards to analyze results.

Algorithm 140

Algorithm Metadata Kafka Systems 140

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. This is what managing data without metadata feels like. This is what managing data without metadata feels like. Effective metadata management is no longer a luxury—it’s a necessity.

Metadata 52

Metadata IT Government High Quality Data 52

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously. Let’s imagine a “Hello, World!”

Kafka 111

Kafka Transportation Metadata Consulting 111

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 98

Management Kafka Metadata Media 98

François Nguyen

JANUARY 18, 2021

In every step,we do not just read, transform and write data, we are also doing that with the metadata. As mentioned in the DataKitchen article, it is deployed automatically with code. As mentioned in the DataKitchen article, it is deployed automatically with code. It is not an extra work by reading the database schema or based on your ETL.

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

LinkedIn Engineering

JANUARY 19, 2024

This multi-entity handover process involves huge amounts of data updating and cloning. Data consistency, feature reliability, processing scalability, and end-to-end observability are key drivers to ensuring business as usual (zero disruptions) and a cohesive customer experience. Push for eventual success of the request.

Recruitment 88

Recruitment Data Process Process Kafka 88

Data Engineering Podcast

NOVEMBER 26, 2023

Developing event-driven pipelines is going to be a lot easier - Meet Functions! Memphis Logo]([link] Developing event-driven pipelines is going to be a lot easier - Meet Functions! Developing event-driven pipelines is going to be a lot easier - Meet Functions! Go to [materialize.com]([link] today and get 2 weeks free!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

JUNE 17, 2023

By incorporating column-level lineage in the data modeling process it encourages a more robust and well-informed design. In this episode Satish Jayanthi explores the benefits of incorporating column-aware tooling in the data modeling process. What are some of the ways that that information can be fed into other contexts?

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

Cloudera

NOVEMBER 13, 2020

Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? Metadata Caching. In the previous design each Impala coordinator daemon kept an entire copy of the contents of the catalog cache in memory and had to be explicitly notified of any external metadata changes.

Metadata 144

Metadata Coding SQL Database 144

The Pragmatic Engineer

AUGUST 3, 2023

To make this migration easier and as seamless as possible, we will give developers the ability to export their deep-link metadata.” Events tech company Hopin selling its events tech business. The ‘startup purge’ event that we predicted at the beginning the year would come seems to, sadly, be here.

Metadata 163

Metadata Engineering Building Technology 163

Netflix Tech

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Process 88

Process Data Pipeline Datasets SQL 88

Cloudyard

JULY 16, 2023

Read Time: 2 Minute, 13 Second During the last post we discussed about the Event tables and how these are used for Error logging in your process. The UDTF was responsible of parsing JSON data in a tabular format and identify any erroneous records and log into an Event table. Event tables.

Metadata 52

Metadata Python Utilities Coding 52

Netflix Tech

NOVEMBER 14, 2023

In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing. We’ll return to the sample customer lifecycle: Processing Requirement : Keep track of the end-of-hour state of accounts, e.g., Active/Upgraded/Downgraded/Canceled.

Metadata 102

Metadata Data Pipeline Scala Data Workflow 102

Data Engineering Podcast

SEPTEMBER 17, 2023

Linked data technologies provide a means of tightly coupling metadata with raw information. what are the characteristics that distinguish a knowledge graph from What are the layers/stages of applications and data that can/should incorporate JSON-LD as the representation for records and events?

Building 189

Building SQL BI Python 189

Maxime Beauchemin

JANUARY 20, 2017

This discipline also integrates specialization around the operation of so called “big data” distributed systems, along with concepts around the extended Hadoop ecosystem, stream processing, and in computation at scale. Sure, there’s a need to abstract the complexity of data processing, computation and storage.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Confluent

AUGUST 1, 2019

We defined the following requirements for the new shared state microservices that would be built: A uniform way to consume events, construct shared state (using varying algorithms), and generate shared state events. About a year ago, we decided to give our shared state scenarios an overhaul that would address these and other concerns.

Kafka 20

Kafka Systems Building Metadata 20

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. So our solution was to start using an intentional contract: Events. What are Events? Events are facts about what happened within your service.

Kafka 52

Kafka ETL Tools BI Database 52

Data Engineering Podcast

MARCH 25, 2023

RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code. Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! Don't miss out on our only event this year!

MySQL 130

MySQL Python Architecture Machine Learning 130

Confluent

MAY 7, 2019

Although it is the simplest way to subscribe to and access events from Kafka, behind the scenes, Kafka consumers handle tricky distributed systems challenges like data consistency, failover and load balancing. Data processing requirements. Every developer who uses Apache Kafka ® has used a Kafka consumer at least once.

Kafka 111

Kafka Accessible Accessibility Metadata 111

Data Engineering Weekly

MAY 16, 2023

The bias toward correctness will increase the processing time, which may not be feasible when speed is a priority. Let’s talk about the data processing types. The industry follows two categories of data processing techniques to handle the trade-off between speed and correctness.

Engineering 112

Engineering Kafka Data Pipeline Data Warehouse 112

François Nguyen

MARCH 22, 2021

We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing. ” He/She is managing triggers, he/she needs to check conditions (event type ? TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality.

Technology 130

Technology Architecture Google Cloud Metadata 130

Netflix Tech

NOVEMBER 28, 2018

Today, we would like to share two additional layers of security: API enforcement and metadata protection. Protecting the EC2 Metadata service so that credentials cannot be retrieved via a vulnerability in an application such as Server Side Request Forgery ( SSRF ).

AWS 99

AWS Metadata Amazon Web Services Cloud 99

Cloudera

AUGUST 10, 2021

We understand that migrating your data platform to the latest version can be an intricate task, and at Cloudera we’ve worked hard to simplify this process for all our customers. . The Atlas – Kafka integration is provided by the Atlas Hook that collects metadata from Kafka and stores it in Atlas.

Cloud 98

Cloud Kafka Metadata SQL 98

Data Engineering Podcast

MAY 18, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. What is the current state of the ecosystem for generating and sharing metadata between systems? What are your goals for the OpenLineage effort?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Netflix Tech

OCTOBER 8, 2024

Building on these foundational abstractions, we developed the TimeSeries Abstraction  — a versatile and scalable solution designed to efficiently store and query large volumes of temporal event data with low millisecond latencies, all in a cost-effective manner across various use cases. For example: {“device_type”: “ios”}.

Bytes 96

Bytes Datasets Metadata Data 96

Confluent

MAY 21, 2019

This leads us to event streaming microservices patterns. Now that the profile change event is published, it can be received by the quote service. There are databases, document stores, data files, NoSQL and ETL processes involved. In fact, schemas are more than just a contract between two event streaming microservices.

Kafka 110

Kafka Insurance Architecture Database 110

Netflix Tech

DECEMBER 14, 2018

NMDB is built to be a highly scalable, multi-tenant, media metadata system that can serve a high volume of write/read throughput as well as support near real-time queries. In NMDB we think of the media metadata universe in units of “DataStores”. This allows for a seamless self-service process for creating and managing a DS.

Media 97

Media Database Metadata Data Schemas 97

Confluent

APRIL 2, 2019

the event streaming platform built by the original creators of Apache Kafka. What do we mean by contextual event-driven applications? Well, infrastructure to support event-driven applications has been around for decades; mere messaging is nothing new. Accelerate the development of contextual event-driven applications.

Kafka 21

Kafka Java Cloud Metadata 21

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. If you missed the event or need a refresh of what was presented, watch any Snowday session on demand. Learn more about Iceberg Tables here. Learn more here.

Metadata 96

Metadata Python AWS Government 96

Confluent

APRIL 11, 2019

Unless a use case actively requires a specific database, companies use S3 for storage and process the data with Amazon Elastic MapReduce (EMR) or Amazon Athena. And no one likes missing events. Kafka clients and the S3 SDK libraries should be all you need to get events from Kafka to S3.

Kafka 110

Kafka AWS Metadata Architecture 110

Scott Logic

APRIL 22, 2024

I’ve coloured the data entities according to their types and we see there’s a few different patterns like events and state which we’ll discuss in a moment. I’ll use the term message in this blog as a generic term for asynchronous communication that may be state or commands or events.

AWS 52

AWS Metadata Systems Kafka 52

ProjectPro

FEBRUARY 8, 2023

For e.g., Finaccel, a leading tech company in Indonesia, leverages AWS Glue to easily load, process, and transform their enterprise data for further processing. It offers a simple and efficient solution for data processing in organizations. Users can schedule ETL jobs, and they can also choose the events that will trigger them.

AWS 98

AWS Metadata Scala Data Lake 98

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. In your experience, what would be required to allow for triggering data processes based on the needs of the data consumers?

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Lyft Engineering

JUNE 28, 2023

While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort. The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. What is your process for establishing and evaluating selection criteria for any new technologies that you adopt?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100