Data Engineering Podcast

FEBRUARY 18, 2024

Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems? Don't miss out on our only event this year!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

dbt Developer Hub

APRIL 20, 2025

We are committed to building the data control plane that enables AI to reliably access structured data from across your entire data lineage. Both AI agents and business stakeholders will then operate on top of LLM-driven systems hydrated by the dbt MCP context. What is MCP? Why does this matter? MCP addresses this challenge.

Structured Data 98

Structured Data SQL BI Project 98

Striim

JANUARY 30, 2025

Todays organizations have access to more data than ever before, and consequently are faced with the challenge of determining how to transform this tremendous stream of real-time information into actionable insights. Encryption, access controls, and regulatory compliance (HIPAA, GDPR, etc.) patient records or geolocation data).

Systems 52

Systems Management Hospitality Healthcare 52

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. Still, I’m puzzled by how long the system has been down. If it was an update to Oracle, or to the operating system, then why not roll back the update?

Banking 271

Banking Utilities Database Engineering 271

Snowflake

NOVEMBER 26, 2024

A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. Processing some 90,000 tables per day, the team oversees the ingestion of more than 100 terabytes of data from upward of 8,500 events daily.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. On the other hand, fail-secure systems focus on maintaining continuous security, keeping doors locked even in difficult conditions to protect assets.

Systems 105

Systems Electronics Hospitality Architecture 105

Snowflake

MAY 16, 2024

Our modern approach accelerates digital transformation, connects previously siloed systems, increases operational efficiencies, and can deliver better outcomes for constituents verifying digital credentials. The post How Snowflake and Merit Helped Provide Over 120,000 Students with Access to Education Funding appeared first on Snowflake.

Education 104

Education Accessible Accessibility Recruitment 104

Snowflake

APRIL 2, 2025

The data warehouse solved for performance and scale but, much like the databases that preceded it, relied on proprietary formats to build vertically integrated systems. Learn more Join Snowflake at Iceberg Summit , a two-day event taking place in San Francisco on April 8 and virtually April 9.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

Engineering at Meta

JANUARY 22, 2025

It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems. It enhances the traceability of data flows within systems, ultimately empowering developers to swiftly implement privacy controls and create innovative products. Hack, C++, Python, etc.)

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

The Pragmatic Engineer

OCTOBER 10, 2024

Wordpress is the most popular content management system (CMS),  estimated  to power around 43% of all websites; a staggering number! We are talking about a competitor to WP Engine (Automattic) with no concrete knowledge of WP Engine’s true revenue, and demanding full access to detailed revenue reports. 25 Sep: Block.

Consulting 347

Consulting AWS Engineering Software Engineer 347

Snowflake

JANUARY 30, 2025

Gen AI makes this all easy and accessible because anyone in an enterprise can simply interact with data by using natural language. What if our app doesnt have access to the right data and generates inaccurate results for stakeholders? But dont worry if you missed it now you can watch the event on demand.

Unstructured Data 97

Unstructured Data Entertainment Healthcare Telecommunication 97

Netflix Tech

MARCH 28, 2025

By Ko-Jen Hsiao , Yesu Feng and Sudarshan Lamkhede Motivation Netflixs personalized recommender system is a complex system, boasting a variety of specialized machine learned models each catering to distinct needs including Continue Watching and Todays Top Picks for You. Refer to our recent overview for more details).

Metadata 84

Metadata Bytes Data Mining Entertainment 84

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.

Systems 94

Systems Media Machine Learning Data Warehouse 94

Netflix Tech

FEBRUARY 14, 2025

It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure. In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily.

Kafka 66

Kafka Datasets Metadata Utilities 66

The Pragmatic Engineer

MARCH 13, 2023

In case you somehow missed it: we went through the fastest bank run in history, in an event that impacted about half of all VC-funded startups in the US and UK. In this special edition, I cover the events of this blow to the tech economy. It’s been a wild weekend, starting Friday. Deposits in Silicon Valley Bank, 1991-2023.

Banking 243

Banking Insurance Portfolio Media 243

Snowflake

APRIL 9, 2025

KAWA Analytics Digital transformation is an admirable goal, but legacy systems and inefficient processes hold back many companies efforts. PTA Robotics PTA Robotics AI-powered vineyard disease prediction system leverages drone imagery, Internet of Things data and weather insights to detect vineyard disease risks before symptoms appear.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. OpenSSL : the cryptography and SSL/TLS toolkit comes with a built-in performance benchmarking capability Lmbench : tools for performance analysis for UNIX/POSIX system. This was one section from last week’s The Pulse.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. In the event of these different cluster errors, what are the strategies for mitigating and recovering from those failures? Operating it at scale, however, is notoriously challenging.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Weekly

OCTOBER 27, 2024

impactdatasummit.com Uber: Streamlining Financial Precision - Uber’s Advanced Settlement Accounting System Possibly one of the complicated pipelines to build is the Financial reconciliation engine. Passes include app-brain-date networking, birds of a feature, post-event parties, etc. What are you waiting for?

Data Engineering 58

Data Engineering Data Engineer Engineering Unstructured Data 58

Cloudera

NOVEMBER 13, 2024

Several LLMs are publicly available through APIs from OpenAI , Anthropic , AWS , and others, which give developers instant access to industry-leading models that are capable of performing most generalized tasks. Data Preparation.

Datasets 87

Datasets Machine Learning Coding Data Preparation 87

Pinterest Engineering

JANUARY 18, 2024

In particular, our machine learning powered ads ranking systems are trying to understand users’ engagement and conversion intent and promote the right ads to the right user at the right time. Specifically, such discrepancies unfold into the following scenarios: Bug-free scenario : Our ads ranking system is working bug-free.

Systems 98

Systems Machine Learning Data Mining Algorithm 98

Engineering at Meta

AUGUST 9, 2023

Explore is one of the largest recommendation systems on Instagram. Using more advanced machine learning models, like Two Towers neural networks, we’ve been able to make the Explore recommendation system even more scalable and flexible. locally popular media), which further contributes to system scalability.

Systems 90

Systems Media Algorithm Machine Learning 90

Netflix Tech

DECEMBER 19, 2024

Many of these projects are under constant development by dedicated teams with their own business goals and development best practices, such as the system that supports our content decision makers , or the system that ranks which language subtitles are most valuable for a specific piece ofcontent.

Machine Learning 81

Machine Learning Project Data Warehouse Coding 81

Data Engineering Weekly

NOVEMBER 3, 2024

Foundation Capital: A System of Agents brings Service-as-Software to life software is no longer simply a tool for organizing work; software becomes the worker itself, capable of understanding, executing, and improving upon traditionally human-delivered services. What are you waiting for? Register for IMPACT today!

Data Engineering 63

Data Engineering Data Engineer Engineering Pipeline-centric 63

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. We used Elasticsearch dashboards to analyze results.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Snowflake

NOVEMBER 11, 2024

Ingest data more efficiently and manage costs For data managed by Snowflake, we are introducing features that help you access data easily and cost-effectively. This reduces the overall complexity of getting streaming data ready to use: Simply create external access integration with your existing Kafka solution.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Precisely

FEBRUARY 6, 2025

Automation and AI are pushing organizations forward but the reality is that the core systems that run our business still exist. While a cloud-first company may not have on-prem legacy systems, most companies are running an IBM Z or IBM i for transactional data processes. Time, effort and labor costs saved. Whats next?

IT 64

IT Management Datasets Systems 64

Data Engineering Weekly

FEBRUARY 18, 2025

Kafka is designed for streaming events, but Fluss is designed for streaming analytics. Kafka is designed to be a black box to collect all kinds of data, so Kafka doesn't have built-in schema and schema enforcement; this is the biggest problem when integrating with schematized systems like Lakehouse. When to use Fluss vs Apache Pinot?

Kafka 74

Kafka Lambda Architecture SQL Architecture 74

Data Engineering Podcast

MARCH 3, 2024

Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! What are the skills and systems that need to be in place to effectively execute on an AI program? "AI" Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today!

Programming 162

Programming Data Lake High Quality Data Machine Learning 162

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

The Pragmatic Engineer

OCTOBER 10, 2023

I wrote code for drivers on Windows, and started to put a basic observability system in place. EC2 had no observability system back then: people would spin up EC2 instances but have no idea whether or not they worked. With my team, we built the basics of what is now called AWS Systems Manager. It was a lot of fun!

Software Engineer 327

Software Engineer Software Engineering Architecture Media 327

Snowflake

FEBRUARY 13, 2025

Furthermore, most vendors require valuable time and resources for cluster spin-up and spin-down, disruptive upgrades, code refactoring or even migrations to new editions to access features such as serverless capabilities and performance improvements.

Management 90

Management Government Cloud Unstructured Data 90

Data Engineering Weekly

MARCH 2, 2025

Get the report → Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. Data Council 2025 is set for April 22-24 in Oakland, CA. The blog provides an excellent analysis of smallpond compared to Spark and Daft.

Data Engineering 63

Data Engineering Data Engineer Engineering Datasets 63

Striim

MARCH 14, 2025

This is why companies are abandoning traditional ETL in favor of Change Data Capture (CDC) and event-driven architectures. Events (deposits and withdrawals) are captured and streamed in real time using change data capture. Enterprises must also solve practical problems to ensure AI models dont have access to customer PII.

Government 52

Government Data Pipeline Data Lake Architecture 52

Data Engineering Podcast

FEBRUARY 25, 2024

Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year! Don't miss out on our only event this year!

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Weekly

FEBRUARY 23, 2025

Try Astro Free → Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. I wonder if these systems expand more capabilities that eventually fall on their own weight. Let me know in the comments.

Data Engineering 63

Data Engineering Data Engineer Engineering Kafka 63

Edureka

APRIL 22, 2025

Enter Amazon EventBridge, a fully managed serverless event bus service that makes it easier to build event-driven applications using data from your AWS services, custom applications, or SaaS providers. It is a fully managed, serverless event bus service that allows applications to communicate with each other using events.

AWS 40

AWS Architecture Media Cloud 40