Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. What are the experimental methods that you are using to gain understanding in the opportunities and practical limits of those systems? What do you have planned for the future of your academic research?

Data Process 147

Data Process Process Data Lake High Quality Data 147

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Engineering at Meta

DECEMBER 10, 2024

Were explaining the end-to-end systems the Facebook app leverages to deliver relevant content to people. At Facebooks scale, the systems built to support and overcome these challenges require extensive trade-off analyses, focused optimizations, and architecture built to allow our engineers to push for the same user and business outcomes.

Systems 76

Systems Architecture Engineering Data Pipeline 76

Engineering at Meta

JUNE 19, 2024

We’re introducing parameter vulnerability factor (PVF) , a novel metric for understanding and measuring AI systems’ vulnerability against silent data corruptions (SDCs) in model parameters. But the growing complexity and diversity of AI hardware systems also brings an increased risk of hardware faults such as bit flips.

Systems 99

Systems Deep Learning Manufacturing Architecture 99

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  Avoiding downtime was nerve-wracking, and the notion of a 'rollback' was as much a relief as a technical process. 

Engineering 302

Engineering Bytes Cloud Computing AWS 302

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineer 363

Precisely

DECEMBER 16, 2024

Modern IT environments require comprehensive data for successful AIOps, that includes incorporating data from legacy systems like IBM i and IBM Z into ITOps platforms. AIOps presents enormous promise, but many organizations face hurdles in its implementation: Complex ecosystems made of multiple, fragmented systems that lack interoperability.

Systems 52

Systems IT Machine Learning Insurance 52

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 246

Hadoop Data Collection Cloud Technology 246

Start Data Engineering

MAY 28, 2024

Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Project demo 3. Use DuckDB 4.4.

Data Pipeline 219

Data Pipeline Python Building Data 219

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineer 240

Data Engineer Data Engineering Engineering Data 240

Analytics Vidhya

JANUARY 31, 2023

It is a powerful resource management system for a horizontal server environment. It is designed to be more flexible and generic than the original Hadoop MapReduce system, making it an attractive choice for companies looking to implement Hadoop. Introduction YARN stands for Yet Another Resource Negotiator.

Hadoop 229

Hadoop Designing Systems Process 229

The Pragmatic Engineer

OCTOBER 17, 2024

We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting:  Netlify : chosen thanks to their super smooth preview system with SSR support. Internal comms: Chat: Slack Coordination / project management: Linear 3.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It is a core component of the Apache Hadoop ecosystem and allows for storing and processing large datasets across multiple commodity servers.

Data Storage 185

Data Storage Big Data Hadoop Datasets 185

The Pragmatic Engineer

FEBRUARY 29, 2024

But first, a few current cases of systems whose developers didn’t: In Sweden, card payments are down at a leading supermarket chain. Airline Avianca printed tickets dated as 3/1 instead of 2/29, thanks to their system not accounting for the leap day. The system was almost fully restored before noon.”

Software Engineer 220

Software Engineer Software Engineering Banking Engineering 220

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Support Data Engineering Podcast Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Support Data Engineering Podcast

Kafka 245

Kafka Data Lake High Quality Data SQL 245

The Pragmatic Engineer

AUGUST 22, 2023

Each project typically takes several years to create, with shifting hardware specifications and emerging competitors and trends to anticipate and react to, during the process. Perhaps unsurprisingly, this whole process was heavily affected by the global health crisis and the need to work from home. Prototype vs final version.

Software Engineer 207

Software Engineer Software Engineering Consulting Entertainment 207

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This technique facilitates validation on multiple fronts.

Utilities 139

Utilities Systems Architecture Coding 139

Jesse Anderson

JUNE 12, 2024

We also discuss the various systems using Kafka’s protocol. When creating streaming systems, we can choose two of the three choices. I think it’s an interesting way of framing the tradeoffs we deal with in streaming systems. There was a chasm between the pub/sub-system and the data lake.

Kafka 147

Kafka Data Lake Architecture Cloud 147

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 140

Algorithm Metadata Kafka Systems 140

Start Data Engineering

MAY 30, 2024

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python 147

Python Data Engineer Data Engineering Engineering 147

Data Engineering Podcast

OCTOBER 29, 2023

In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection. What is driving the broad adoption of columnar stores as a separate environment from transactional systems? Datafold : ![Datafold]([link]

Database 189

Database BI SQL Machine Learning 189

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 176

Recruitment Java Coding Project 176

The Pragmatic Engineer

AUGUST 10, 2023

The period from April to mid-May was challenging: I found myself in hiring freezes and canceled processes. Simplify helped me a lot to automatically fill applications in the most popular applicant tracking systems (ATS.) ’ How did you find the interview processes?

Engineering 196

Engineering Recruitment Retail Software Engineer 196

The Pragmatic Engineer

APRIL 6, 2023

For AI,  we’ve built a system to efficiently use GPT-4 for this purpose, including auto-crafting prompts and performing pre and post-processing. Our system is using purely Serverless to process the data. During processing, we match companies, titles and more, with our dataset.

Software Engineer 194

Software Engineer Software Engineering Datasets Database 194

The Pragmatic Engineer

APRIL 19, 2023

For "jargon architects," this tends to happen because engineers assume that as they don't understand the jargon, they must also not understand the thought process, so do not challenge them. I was sceptical that any system would automatically reject resumes, because I never saw this as a hiring manager.

Software Engineer 295

Software Engineer Software Engineering Engineering Media 295

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Podcast

MAY 21, 2023

Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. Stream processing technologies have been around for around a decade.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Jesse Anderson

OCTOBER 5, 2023

In another post , I introduced a concept from Justin Coffey about how much systems can be changed from their original design. Teams must know this, or you’ll have vendor lock-in on an ostensibly open-source system. Flink’s batch processing is improving, and there are some caveats to know. million in cash and another $52.3

Kafka 195

Kafka Finance Cloud Designing 195

Engineering at Meta

AUGUST 27, 2024

At Meta, we’ve been diligently working to incorporate privacy into different systems of our software stack over the past few years. Purpose limitation, a core data protection principle, is about ensuring data is only processed for explicitly stated purposes.

Programming Language 122

Programming Language Coding Data Warehouse Systems 122

Data Engineering Podcast

MARCH 3, 2024

Who needs to be involved in the process of defining and developing that program? What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" What are the most interesting, unexpected, or challenging lessons that you have learned while working on powering AI systems?

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Precisely

SEPTEMBER 23, 2024

More specifically, these companies use IBM Z systems. This growth is driven by the ongoing need for reliable, high-capacity computing systems, particularly in sectors like banking, finance, and government​. An IBM Z System mainframe can survive an earthquake having a magnitude of at least 8.0 during the period from 2024 to 2032.

Banking 116

Banking Insurance Finance Systems 116

Maxime Beauchemin

APRIL 26, 2023

To support engineers in this process, I am introducing Promptimize , a Python toolkit designed to measure success and outcomes while doing prompt engineering. This process is essential for developing scalable solutions and optimizing prompts across diverse use cases. What a stellar assistant!

SQL 148

SQL Database Engineering Software Engineer 148

Engineering at Meta

OCTOBER 24, 2023

As an example, we looked at Moments , the photo sharing app Meta launched in 2015 and eventually shut down in 2019, and how SCARF can help with the deprecation process through its workflow management capabilities. It leverages this analysis to submit change requests to remove this code from our systems.

Coding 142

Coding Programming Language Python MySQL 142