Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. In the event of these different cluster errors, what are the strategies for mitigating and recovering from those failures? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

The Pragmatic Engineer

MAY 25, 2023

Glassdoor could make the process a lot clearer by publishing a moderation log which details when and why it removed a review. Organize a “Glassdoor review event,” asking employees to leave honest reviews. This log could contain only the redacted parts of affected reviews to ensure the terms of service are not broken.

Software Engineer 316

Software Engineer Software Engineering AWS Engineering 316

The Pragmatic Engineer

JUNE 1, 2023

for the simulation engine Go on the backend PostgreSQL for the data layer React and TypeScript on the frontend Prometheus and Grafana for monitoring and observability And if you were wondering how all of this was built, Juraj documented his process in an incredible, 34-part blog series. You can read this here. Serving a web page.

Education 363

Education Project PostgreSQL Software Engineer 363

Data Engineering Podcast

MARCH 3, 2024

Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Who needs to be involved in the process of defining and developing that program? Data Council Logo]([link] Join us at the top event for the global data community, Data Council Austin.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

The Pragmatic Engineer

OCTOBER 31, 2023

” And an update inside the final hour and a half of the outage: “2:00 PM PDT Many AWS services are now fully recovered and marked Resolved on this event. Lambda is working to process these messages during the next few hours and during this time, we expect to see continued delays in the execution of asynchronous invocations.

AWS 232

AWS Google Cloud Cloud Engineering 232

The Pragmatic Engineer

MARCH 3, 2023

I asked Googlers the reason why these events have been canceled and one thing became clear: most of the program managers who worked on the coding competitions were recently let go in Google’s historic job cuts. In the beginning of February, Google announced the delays in the registration process.

Coding 176

Coding IT Software Engineer Software Engineering 176

Data Engineering Podcast

MAY 21, 2023

Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

KDnuggets

DECEMBER 20, 2021

Many real-world datasets consist of records of events that occur at arbitrary and irregular intervals. These datasets then need to be processed into regular time series for further analysis. We will use the AI & Analytics Engine to illustrate how you can prepare your time-series data in just 1 step.

Engineering 158

Engineering Datasets Data Process 158

Zalando Engineering

NOVEMBER 8, 2023

Postgres Logical Replication at Zalando Builders at Zalando have access to a low-code solution that allows them to declare event streams that source from Postgres databases. At the time of writing, there are hundreds of these Postgres-sourced event streams out in the wild at Zalando. Simple, right?

PostgreSQL 138

PostgreSQL Java Database Bytes 138

The Pragmatic Engineer

OCTOBER 17, 2024

Code and raw data repository:   Version control: GitHub Heavily using GitHub Actions for things like getting warehouse data from vendor APIs, starting cloud servers, running benchmarks, processing results, and cleaning up after tuns. Internal comms: Chat: Slack Coordination / project management: Linear 3.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

KDnuggets

JUNE 24, 2022

We explore time series processing, anomaly detection, and an event-based view of the news. In this article we introduce a few tools and techniques for studying relationships between the stock market and the news.

Coding 137

Coding Process Data 137

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. This process can also be used to track the provenance of increments.

Datasets 101

Datasets Computer Science Systems Kafka 101

Data Engineering Podcast

MAY 28, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. When there is no internal data talent to assist with hiring, what are some of the problems that manifest in the hiring process?

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

Data Engineering Podcast

APRIL 16, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. What was your process for defining the problem you are focused on solving and the method to achieve that outcome? Business intelligence is a crowded market.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

FEBRUARY 25, 2024

When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow ecosystem ready and waiting with useful building blocks to accelerate the process. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Promo Code: dataengpod20 Dagster : ![Dagster

Database 162

Database Technology Data Lake High Quality Data 162

The Pragmatic Engineer

NOVEMBER 21, 2023

Avoiding downtime was nerve-wracking, and the notion of a 'rollback' was as much a relief as a technical process.  After this zero-byte file was deployed to prod, the Apache web server processes slowly picked up the empty configuration file. Our deployments were initially manual. Apache started to log like a maniac.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Don't miss out on our only event this year!

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

APRIL 9, 2023

Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

The Pragmatic Engineer

JULY 20, 2023

Venture funding is on a downward trend , and we seem to be at the start – or the middle – of a “startup purge” event. This news is hot off the press, publicly announced by Postman and by Akita yesterday, and you are among the early ones to hear about this event. On hiring Every startup hires differently.

Building 208

Building Programming Language Programming Python 208

Confluent

OCTOBER 4, 2024

Bad data causes serious issues and outages for downstream data users. It can be prevented with good data practices, but it must be properly fixed when it does occur.

Data 72

Data IT Process 72

Data Engineering Podcast

NOVEMBER 26, 2023

Developing event-driven pipelines is going to be a lot easier - Meet Functions! Memphis Logo]([link] Developing event-driven pipelines is going to be a lot easier - Meet Functions! Developing event-driven pipelines is going to be a lot easier - Meet Functions! Go to [materialize.com]([link] today and get 2 weeks free!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

DoorDash Engineering

MARCH 27, 2024

Real-time event processing is a critical component of a distributed system’s scalability. At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. In Kafka, a test tenant processing production event can cause data inconsistencies, including outages and other incidents.

Kafka 103

Kafka Architecture Algorithm Systems 103

Data Engineering Podcast

MARCH 10, 2024

Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! How does the inclusion of Nessie in a data lake influence the overall workflow of developing/deploying/evolving processing flows? Don't miss out on our only event this year!

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Data Engineering Weekly

MAY 16, 2023

The bias toward correctness will increase the processing time, which may not be feasible when speed is a priority. Let’s talk about the data processing types. The industry follows two categories of data processing techniques to handle the trade-off between speed and correctness.

Engineering 112

Engineering Kafka Data Pipeline Data Warehouse 112

Snowflake

JUNE 10, 2024

Discovering and surfacing telemetry traditionally can be a tedious and challenging process, especially when it comes to pinpointing specific issues for debugging. A default Event Table (public preview soon) is in the Snowflake database of every account, removing the need to create and manage your own custom event table.

Python 107

Python Java Hadoop Coding 107

Netflix Tech

JUNE 1, 2023

New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. Next, we launched a Mantis job that processed all requests in the stream and replayed them in a duplicate production environment created for replay traffic. We used Elasticsearch dashboards to analyze results.

Algorithm 140

Algorithm Metadata Kafka Systems 140

Data Engineering Podcast

MARCH 25, 2023

RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code. Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! Don't miss out on our only event this year!

MySQL 130

MySQL Python Architecture Machine Learning 130

Robinhood

JUNE 6, 2024

Robinhood and Bitstamp customers can expect the same level of service, security and reliability and as we move forward, we are committed to maintaining transparency throughout this process. Robinhood expects the final deal consideration to be approximately $200 million in cash, subject to customary purchase price adjustments.

Retail 129

Retail Systems Process Management 129

Netflix Tech

SEPTEMBER 10, 2024

One issue that often complicates this process is the "noisy neighbor" problem. The sched_wakeup and sched_wakeup_new hooks are invoked when a process changes state from 'sleeping' to 'runnable.' ' They let us identify when a process is ready to run and is waiting for CPU time.

Utilities 100

Utilities Coding Programming Java 100

Knowledge Hut

MAY 3, 2024

A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. The demand for stream processing is increasing every day in today’s era. Kafka streams provide true a-record-at-a-time processing capabilities.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Data Engineering Podcast

JUNE 17, 2023

By incorporating column-level lineage in the data modeling process it encourages a more robust and well-informed design. In this episode Satish Jayanthi explores the benefits of incorporating column-aware tooling in the data modeling process. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Let’s dive in! What is late-arriving data? How does late-arriving data impact us?

Data Engineering 109

Data Engineering Data Engineer Engineering Metadata 109

Cloudera

MAY 10, 2024

It accelerates the entire data lifecycle from streaming and ingestion to processing, analytics, and AI. This enables unbiased, accelerated innovation and, for the first time, provides a common standard for all data in the organization, irrespective of the processing engine.

Government 109

Government Data Governance Data Pipeline Big Data 109

Data Engineering Podcast

MAY 7, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. The best problem to have.but a lot of "sweat smiles emojis going around" Onboarding requires requires a one-time backfill of all historical events in time.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Christophe Blefari

MARCH 24, 2023

I really liked the presentations from Mage and Kestra and even if I was focus on hosting the event it was great to see 2 other visions about the future of orchestration. Ludovic, the CTO who presented Kestra at the event, said that he started the development while at a mission at Leroy Merlin where people were heavily unhappy about Airflow.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Netflix Tech

JULY 22, 2024

This is useful if a workflow is designed to always process the latest data, such as processing the latest snapshot of an entire table each time. This helps to fan out and distribute the execution, enabling the processing of large amounts of data within the time limit. An example of a timeline is included here for reference.

Machine Learning 104

Machine Learning Coding Data Pipeline Java 104

Knowledge Hut

JANUARY 16, 2024

” ― Denis Waitley What are SAFe Agile Events (or) Ceremonies? The key link between these levels is the SAFe-specific events which help with smooth value delivery facilitation. The events help with alignment across teams, ARTs, etc thus helping in managing risk by providing a level-based cadence and synchronization.

Portfolio 97

Portfolio Business Analyst Programming Certification 97