Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services and systems need to be kept up to date, but so does the code that controls their behavior.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

JANUARY 21, 2019

Summary Building internal expertise around big data in a large organization is a major competitive advantage. They discuss the challenges of being at global scale from the start, hiring and training talented engineers, prototyping and deploying new systems in the cloud, and what they have learned in the process.

Big Data 100

Big Data Systems Building Hadoop 100

Data Engineering Podcast

JULY 27, 2020

Summary A majority of the scalable data processing platforms that we rely on are built as distributed systems. Kyle Kingsbury created the Jepsen framework for testing the guarantees of distributed data processing systems and identifying when and why they break. This brings with it a vast number of subtle ways that errors can creep in.

Systems 100

Systems Building Scala Java 100

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.

Systems 93

Systems Media Machine Learning Data Warehouse 93

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Monte Carlo

FEBRUARY 23, 2024

And this renewed focus on data quality is bringing much needed visibility into the health of technical systems. Their Snowpark Container Services enable developers to register and deploy containerized data apps using secure Snowflake-managed infrastructure with configurable hardware options, like accelerated computing with NVIDIA GPUs.

Systems 64

Systems Building Retail Data Lake 64

Edureka

FEBRUARY 20, 2023

Principles of Quality Management System: Quality planning, assurance, control, and improvement are the four primary parts of quality management. To hold them to a high and consistent level, quality management monitors the activities, tasks, and processes (inputs) utilised to produce a product or service (outputs).

Systems 52

Systems Management Manufacturing Food 52

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. As they scale, the problems of visibility and dependency management can increase at an exponential rate. Interview Introduction How did you get involved in the area of data management?

Systems 100

Systems Metadata Building MongoDB 100

Netflix Tech

FEBRUARY 18, 2022

To this end, we developed a Rapid Event Notification System (RENO) to support use cases that require server initiated communication with devices in a scalable and extensible manner. In this blog post, we will give an overview of the Rapid Event Notification System at Netflix and share some of the learnings we gained along the way.

Systems 133

Systems Architecture Portfolio Designing 133

Lyft Engineering

APRIL 25, 2023

Building a large scale unsupervised model anomaly detection system — Part 2 Building ML Models with Observability at Scale By Rajeev Prabhakar , Han Wang , Anindya Saha Photo by Octavian Rosca on Unsplash In our previous blog we discussed the different challenges we faced for model monitoring and our strategy for addressing some of these problems.

Systems 76

Systems Building Machine Learning Datasets 76

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Are you tired of dealing with the headache that is the 'Modern Data Stack'? We feel your pain.

Building 147

Building Machine Learning Metadata Data Integration 147

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Data Engineering Podcast

JULY 5, 2021

Summary At the core of every data pipeline is an workflow manager (or several). Deploying, managing, and scaling that orchestration can consume a large fraction of a data team’s energy so it is important to pick something that provides the power and flexibility that you need.

Systems 100

Systems Management Data Warehouse Programming Language 100

Monte Carlo

FEBRUARY 23, 2024

And this renewed focus on data quality is bringing much needed visibility into the health of technical systems. Their Snowpark Container Services enable developers to register and deploy containerized data apps using secure Snowflake-managed infrastructure with configurable hardware options, like accelerated computing with NVIDIA GPUs.

Systems 52

Systems Building Retail Data Lake 52

Rockset

MARCH 28, 2024

Overview In this guide, you will: Gain a high-level understanding of vectors, embeddings, vector search, and vector databases, which will clarify the concepts we will build upon. Build a dynamic web application using vanilla CSS, HTML, JavaScript, and Flask, seamlessly integrating with the Rockset API and the OpenAI API.

Systems 52

Systems Building Database Utilities 52

RandomTrees

NOVEMBER 5, 2024

Good data management in Salesforce provides insight into improving interactions with customers and making business operations easier. In this blog, we will take you through the basic steps that help in managing data effectively in Salesforce. Step 2: Data Entry and Management 2.1. Step 1: Understand Your Data Model 1.1

Management 52

Management Data Utilities Data Validation 52

Engineering at Meta

MAY 22, 2024

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability, promotes reusability, and improves engineering efficiency. In many cases, it forced our engineers to reinvent the wheel, duplicating work and reducing our ability to quickly adapt systems as requirements evolved.

Data Management 91

Data Management Management Data SQL 91

Monte Carlo

AUGUST 17, 2022

Over the years, I’ve helped companies of all sizes build and maintain data systems—from my days as a data engineer at Facebook to my current role as an end-to-end data solutions consultant. It’s never too early to start building reliability into your data system. The stakes also get higher.

Systems 52

Systems Building Data Data Engineering 52

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. What do you have planned for the future of Scanner?

Data Lake 147

Data Lake Building High Quality Data AWS 147

Cloudera

NOVEMBER 13, 2024

This acquisition delivers access to trusted data so organizations can build reliable AI models and applications by combining data from anywhere in their environment. Additionally, multiple copies of the same data locked in proprietary systems contribute to version control issues, redundancies, staleness, and management headaches.

Metadata 82

Metadata Management Data Governance Government 82

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? What are the different methods that you have used for managing database instances? Why Postgres?

Database 130

Database Building PostgreSQL BI 130

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. These two distinct approaches offer unique strategies for securing buildings and valuables. What are Fail-Safe Door Locks?

Systems 105

Systems Electronics Hospitality Architecture 105

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode Ian Schweer shares his experiences at Riot Games supporting player-focused features such as machine learning models and recommeder systems that are deployed as part of the game binary. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Knowledge Hut

DECEMBER 28, 2023

The information system is a very vast concept that encompasses several aspects like database management, the communication system, various devices, several connections, the internet, collection, organization, and storing data and other information-related applications that are typically used in a business forum.

Systems 52

Systems Telecommunication Technology Certification 52

Lyft Engineering

APRIL 3, 2023

At times, recommendations are also leveraged to manage the marketplace, making sure there’s a healthy balance between ride demand and driver supply. This blog post focuses on the scope and the goals of the recommendation system, and explores some of the most recent changes the Rider team has made to better serve Lyft’s riders.

Systems 87

Systems Pipeline-centric Machine Learning Transportation 87

Zalando Engineering

OCTOBER 16, 2024

Introduction Context and Purpose Our team is part of the Transport teams within the Logistics department, where we build and manage software for internal users, including finance teams, warehouses, and, in the future, our third-party partners. Modularity was key to enabling independent feature development and deployment across teams.

Building 68

Building Transportation Coding Designing 68

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. In this episode Gleb Mezhanskiy shares some valuable advice and insights into how you can build reliable and well-tested data assets with dbt and data-diff.

Project 130

Project Building Data Lake Machine Learning 130

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Does it sound familiar?

Systems 78

Systems Raw Data Metadata Data Cleanse 78

Zalando Engineering

MAY 13, 2024

As a design system evolves alongside with the brand it represents, there are often multiple occasions when a need to introduce variations arises. The previous article on this blog gives a wider overview of the Zalando Design System. The previous article on this blog gives a wider overview of the Zalando Design System.

Designing 59

Designing Systems Architecture Coding 59

Data Engineering Podcast

DECEMBER 2, 2018

Summary Distributed systems are complex to build and operate, and there are certain primitives that are common to a majority of them. Rather then re-implement the same capabilities every time, many projects build on top of Apache Zookeeper. What are the distributed systems primitives that are built into Zookeeper?

Systems 100

Systems Building Kafka Java 100

Rockset

SEPTEMBER 2, 2022

I recently had the good fortune to host a small-group discussion on personalization and recommendation systems with two technical experts with years of experience at FAANG and other web-scale companies. Nikhil Garg is CEO and co-founder of Fennel AI , a startup working on building the future of real-time machine learning infrastructure.

Systems 52

Systems Building Machine Learning NoSQL 52

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake.

Business Intelligence 147

Business Intelligence Building Data Lake BI 147