Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. How have the requirements of generative AI shifted the demand for streaming data systems? Can you describe how Datorios is implemented?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Precisely

MARCH 3, 2025

Data and process automation used to be seen as luxury but those days are gone. Lets explore the top challenges to data and process automation adoption in more detail. Almost half of respondents (47%) reported a medium level of automation adoption, meaning they currently have a mix of automated and manual SAP processes.

Process 59

Process Government Data Finance 59

DataKitchen

MARCH 20, 2025

Unlocking Data Team Success: Are You Process-Centric or Data-Centric? We’ve identified two distinct types of data teams: process-centric and data-centric. Process-centric data teams focus their energies predominantly on orchestrating and automating workflows. They work in and on these pipelines.

Pipeline-centric 64

Pipeline-centric Database-centric Process Data 64

Precisely

NOVEMBER 7, 2024

Key Takeaways: In the face of ransomware attacks, a resilience strategy for IBM i systems must include measures for prevention, detection, and recovery. No platform is immune, not even the reliable and secure IBM i systems. So, how can you keep your IBM i systems resilient even as ransomware risks are on the rise?

Systems 59

Systems Accessibility Accessible Programming 59

Edureka

APRIL 10, 2025

When you hear the term System Hacking, it might bring to mind shadowy figures behind computer screens and high-stakes cyber heists. In this blog, we’ll explore the definition, purpose, process, and methods of prevention related to system hacking, offering a detailed overview to help demystify the concept.

Systems 40

Systems Education Banking Accessibility 40

Start Data Engineering

JANUARY 20, 2025

Guide the interviewer through the process 2.1. Introduction 2. Requirements gathering] Make sure you clearly understand the requirements & business use case 2.2. Understand source data] Know what you have to work with 2.3. Model your data] Define data models for historical analytics 2.4.

Designing 130

Designing Systems Data Engineering Data Engineer 130

Striim

MARCH 21, 2025

What is Real-Time Stream Processing? To access real-time data, organizations are turning to stream processing. To access real-time data, organizations are turning to stream processing. There are two main data processing paradigms: batch processing and stream processing.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Engineering at Meta

JANUARY 22, 2025

It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems. It enhances the traceability of data flows within systems, ultimately empowering developers to swiftly implement privacy controls and create innovative products.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. What are the experimental methods that you are using to gain understanding in the opportunities and practical limits of those systems? What do you have planned for the future of your academic research?

Data Process 147

Data Process Process Data Lake High Quality Data 147

Striim

JANUARY 30, 2025

When integrated effectively, AI and machine learning (ML) models can process data streams at near-zero latency, empowering teams to make split-second decisions. Systems must be capable of handling high-velocity data without bottlenecks. Thats where real-time artificial intelligence (AI) can help.

Systems 52

Systems Management Hospitality Healthcare 52

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  Avoiding downtime was nerve-wracking, and the notion of a 'rollback' was as much a relief as a technical process. 

Engineering 345

Engineering Bytes Cloud Computing AWS 345

Netflix Tech

NOVEMBER 12, 2024

Failures in a distributed system are a given, and having the ability to safely retry requests enhances the reliability of the service. Implementing idempotency would likely require using an external system for such keys, which can further degrade performance or cause race conditions.

Datasets 100

Datasets Computer Science Systems Kafka 100

Simon Späti

MARCH 11, 2025

Therefore, you’ve probably come across terms like OLAP (Online Analytical Processing) systems, data warehouses, and, more recently, real-time analytical databases. But data volumes grow, analytical demands become more complex, and Postgres stops being enough.

Database 130

Database Data Warehouse Data Engineer Data Engineering 130

Jesse Anderson

FEBRUARY 11, 2025

Semih is a researcher and entrepreneur with a background in distributed systems and databases. He then pursued his doctoral studies at Stanford University, delving into the complexities of database systems. Dont forget to subscribe to my YouTube channel to get the latest on Unapologetically Technical!

Computer Science 130

Computer Science Database Design Software Engineering Software Engineer 130

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 297

Recruitment Java Coding Project 297

dbt Developer Hub

APRIL 20, 2025

Both AI agents and business stakeholders will then operate on top of LLM-driven systems hydrated by the dbt MCP context. Todays system is not a full realization of the vision in the posts shared above, but it is a meaningful step towards safely integrating your structured enterprise data into AI workflows. Why does this matter?

Structured Data 98

Structured Data SQL BI Project 98

The Pragmatic Engineer

OCTOBER 17, 2024

We recently covered how CockroachDB joins the trend of moving from open source to proprietary and why Oxide decided to keep using it with self-support , regardless Web hosting:  Netlify : chosen thanks to their super smooth preview system with SSR support. Internal comms: Chat: Slack Coordination / project management: Linear 3.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Analytics Vidhya

MARCH 5, 2023

Introduction Microsoft Azure HDInsight(or Microsoft HDFS) is a cloud-based Hadoop Distributed File System version. A distributed file system runs on commodity hardware and manages massive data collections. It is a fully managed cloud-based environment for analyzing and processing enormous volumes of data.

Hadoop 254

Hadoop Data Collection Cloud Technology 254

Snowflake

NOVEMBER 6, 2024

Alberta Health Services ER doctors automate note-taking to treat 15% more patients The integrated health system of Alberta, Canada’s third-most-populous province, with 4.5 But getting a handle on all the emails, calls and support tickets had historically been a tedious and largely manual process. Cortex is doing a great job for us.”

Hospitality 97

Hospitality Medical Government Software Engineer 97

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 364

Education Project PostgreSQL Software Engineering 364

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. And who better to learn from than the tech giants who process more data before breakfast than most companies see in a year?

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

Netflix Tech

MARCH 28, 2025

By Ko-Jen Hsiao , Yesu Feng and Sudarshan Lamkhede Motivation Netflixs personalized recommender system is a complex system, boasting a variety of specialized machine learned models each catering to distinct needs including Continue Watching and Todays Top Picks for You. Refer to our recent overview for more details).

Metadata 84

Metadata Bytes Data Mining Entertainment 84

Start Data Engineering

MAY 28, 2024

Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Distributed systems are scalable, resilient to failures, & designed for high availability 4.5. Project demo 3. Use DuckDB 4.4.

Data Pipeline 219

Data Pipeline Python Building Data 219

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineering 248

Data Engineering Data Engineer Engineering Data 248

Christophe Blefari

JANUARY 11, 2025

AI companies are aiming for the moon—AGI—promising it will arrive once OpenAI develops a system capable of generating at least $100 billion in profits. Meaning: a YAML configuration system for ingestion and transformations, and now, visualisation with BI-as-code. Meanwhile, the AI landscape remains unpredictable.

Data 130

Data Data Warehouse Coding Programming Language 130

Snowflake

NOVEMBER 26, 2024

A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. Processing some 90,000 tables per day, the team oversees the ingestion of more than 100 terabytes of data from upward of 8,500 events daily.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Engineering at Meta

APRIL 28, 2025

Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale. Specifically, we have adopted a “shift-left” approach, integrating data schematization and annotations early in the product development process.

Metadata 64

Metadata Data Utilities Data Warehouse 64

Analytics Vidhya

MARCH 4, 2025

But processing large-scale data across different systems is often slow. Constant format conversions add processing time and memory overhead. Data is at the core of everything, from business decisions to machine learning. Traditional row-based storage formats struggle to keep up with modern analytics.

Machine Learning 100

Machine Learning Systems Process Data 100

Analytics Vidhya

JANUARY 31, 2023

It is a powerful resource management system for a horizontal server environment. It is designed to be more flexible and generic than the original Hadoop MapReduce system, making it an attractive choice for companies looking to implement Hadoop. Introduction YARN stands for Yet Another Resource Negotiator.

Hadoop 236

Hadoop Designing Systems Management 236

Snowflake

MARCH 6, 2025

Customer intelligence teams analyze reviews and forum comments to identify sentiment trends, while support teams process tickets to uncover product issues and inform gaps in a product roadmap. As data volumes grow and AI automation expands, cost efficiency in processing with LLMs depends on both system architecture and model flexibility.

Unstructured Data 97

Unstructured Data Medical Media Data Workflow 97

Engineering at Meta

FEBRUARY 4, 2025

Here we explore initial system designs we considered, an overview of the current architecture, and some important principles Meta takes into account in making data accessible and easy to understand. We also considered caching data logs in an online system capable of supporting a range of indexed per-user queries. What are data logs?

Accessible 80

Accessible Accessibility Raw Data Data Warehouse 80

Snowflake

APRIL 13, 2025

Agentic AI refers to AI systems that act autonomously on behalf of their users. These systems make decisions, learn from interactions and continuously improve without constant human intervention. Manual processes can be time-consuming and error-prone. What is agentic AI? Scalability As businesses grow, so does their data.

Data Management 78

Data Management Management Consulting Unstructured Data 78

Snowflake

APRIL 16, 2025

Process all your data where it already lives Fragmented data environments and complex cloud architectures impede efficiency and innovation. For streamlining manual processes : Online retailers and food delivery platforms use Cortex AI to automate image descriptions for meals and groceries, reducing manual effort.

Data Analysis 75

Data Analysis Unstructured Data Manufacturing Retail 75

Snowflake

APRIL 9, 2025

KAWA Analytics Digital transformation is an admirable goal, but legacy systems and inefficient processes hold back many companies efforts. PTA Robotics PTA Robotics AI-powered vineyard disease prediction system leverages drone imagery, Internet of Things data and weather insights to detect vineyard disease risks before symptoms appear.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Cloudera

NOVEMBER 13, 2024

Additionally, multiple copies of the same data locked in proprietary systems contribute to version control issues, redundancies, staleness, and management headaches. This guarantees data quality and automates the laborious, manual processes required to maintain data reliability.

Metadata 90

Metadata Management Data Governance Government 90

The Pragmatic Engineer

FEBRUARY 29, 2024

But first, a few current cases of systems whose developers didn’t: In Sweden, card payments are down at a leading supermarket chain. Airline Avianca printed tickets dated as 3/1 instead of 2/29, thanks to their system not accounting for the leap day. The system was almost fully restored before noon.”

Software Engineering 281

Software Engineering Software Engineer Banking Engineering 281