Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Netflix Tech

MARCH 14, 2023

We build creator tooling to enable these colleagues to focus their time and energy on creativity. We implemented a batch processing system for users to submit their requests and wait for the system to generate the output. This limited pilot system greatly reduced the time spent by our users to manually analyze the content.

Media 119

Media Building Algorithm Machine Learning 119

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Cloudera

DECEMBER 1, 2023

It focuses on five key pillars: investing in research and development; unleashing government AI resources; setting standards and policy; building the AI workforce; and advancing trust and security. AI systems must operate within a framework that promotes ethical practices, transparency, and accountability. million), among others.

Building 110

Building Government Transportation Data Governance 110

Pinterest Engineering

JANUARY 4, 2024

In 2020, anticipating the growing needs of the business and to simplify our storage offerings, we decided to consolidate our different key-value systems in the company into a single unified service called KVStore. In order to build a distributed and replicated service using RocksDB, we built a real time replicator library: Rocksplicator.

Database 95

Database Building Datasets Relational Database 95

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. How do you manage the personalization of the AI functionality in your system for each user/team? Your first 30 days are free!

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Tweag

JULY 26, 2023

The vast majority of the Rust projects are using Cargo as a build tool. Cargo is great when you are developing and packaging a single Rust library or application, but when it comes to a fast-growing and complex workspace, one could be attracted to the idea of using a more flexible and scalable build system.

Building 93

Building Metadata Coding Project 93

LinkedIn Engineering

DECEMBER 20, 2023

At LinkedIn, trust is the cornerstone for building meaningful connections and professional relationships. In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform.

Building 91

Building Algorithm Kafka Machine Learning 91

Cloudyard

DECEMBER 24, 2024

In this blog, well explore Building an ETL Pipeline with Snowpark by simulating a scenario where commerce data flows through distinct data layersRAW, SILVER, and GOLDEN.These tables form the foundation for insightful analytics and robust business intelligence. Create a fact table to summarize daily sales.

Building 52

Building Raw Data Scala Business Intelligence 52

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 138

Utilities Systems Architecture Coding 138

Pinterest Engineering

JULY 15, 2024

David Chang; Staff Software Engineer | Develocity, formerly known as Gradle Enterprise, is a powerful tool that speeds up local and CI build time, helps troubleshoot your builds, and analyzes your data. At Pinterest, we have a dedicated team, Mobile Builds, and we ensure that developers can build fast and often.

Building 66

Building Metadata Software Engineer Software Engineering 66

Cloudera

OCTOBER 31, 2023

However, it remains difficult to quickly build these capabilities given the challenges with finding readily available talent and resources to get started rapidly on the AI journey. Customers can quickly and easily build generative AI applications using these new features available in Cloudera.

Building 105

Building Amazon Web Services AWS Machine Learning 105

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Your first 30 days are free!

Database 162

Database Technology Data Lake High Quality Data 162

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. Replay traffic enabled us to test our new systems and algorithms at scale before launch, while also making the traffic as realistic as possible.

Algorithm 139

Algorithm Metadata Kafka Systems 139

databricks

AUGUST 23, 2023

To build customer support bots, internal knowledge graphs, or Q&A systems, customers often use Retrieval Augmented Generation (RAG) applications which leverage pre-trained models.

Building 97

Building Systems 97

databricks

DECEMBER 10, 2024

Inspiration Going on vacation is an enjoyable experience, but planning the trip can take time and effort for most people. There are numerous.

Systems 52

Systems Building Entertainment Media 52

Engineering at Meta

OCTOBER 23, 2023

To optimize build and performance, we developed our own build system called Buck , which was first open-sourced in 2013. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Meta has a very large monorepo, with many different programming languages.

Programming Language 117

Programming Language Java Python Programming 117

DoorDash Engineering

FEBRUARY 27, 2024

We reviewed the architecture of our global search at DoorDash in early 2022 and concluded that our rapid growth meant within three years we wouldn’t be able to scale the system efficiently, particularly as global search shifted from store-only to a hybrid item-and-store search experience. latency reduction and a 75% hardware cost decrease.

Engineering 130

Engineering Systems Designing Architecture 130

Snowflake

DECEMBER 11, 2023

While digital advertisers and paid media professionals are on the hook to build ample sales pipeline and maximize return on ad spend (ROAS), they’re also expected to deliver personalized advertising content while navigating evolving privacy requirements and adhering to consumer expectations—all while extracting insights from siloed ad platforms.

Building 92

Building Cloud Media Algorithm 92

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. However, there are some problems with step (4): The build is not properly isolated: it can (in theory) fetch arbitrary things from the network, access arbitrary files on the filesystem, and even modify other packages.

Project 144

Project Programming Language Building Systems 144

Netflix Tech

JUNE 13, 2023

This is where large-scale system migrations come into play. Our previous blog post presented replay traffic testing — a crucial instrument in our toolkit that allows us to implement these transformations with precision and reliability. Canaries and sticky canaries are valuable tools in the system migration process.

Systems 114

Systems Utilities Entertainment Coding 114

Data Engineering Weekly

JULY 21, 2024

The author highlights key lessons learning building an analytical org from scratch with key highlights, 1. Build reliable data, maintain usable data models, and ensure the data is interpreted correctly for decision-making. The blog is an excellent summary of what one needs to know about Gen-AI to start.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. The blog is an excellent summarization of the common patterns emerging in GenAI platforms.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Tweag

FEBRUARY 8, 2023

Local remote builder macOS is a great desktop environment, but as soon as you move beyond your machine, be it to a remote server, VM or container, you most likely end up in Linux, which is a different system. As for linuxkit-nix , Gabriella Gonzales wrote in her blog post a great overview of how it is lacking in different areas.

Systems 103

Systems Building Cloud IT 103

Data Engineering Podcast

APRIL 14, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. Your host is Tobias Macey and today I'm interviewing Oren Eini about the work of designing and building a NoSQL database engine Interview Introduction How did you get involved in the area of data management? Your first 30 days are free!

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

MAY 7, 2023

The challenge is that most companies have a multitude of systems that contain fragments of the customer's interactions and stitching that together is complex and time consuming. Segment created the Unify product to reduce the burden of building a comprehensive view of customers and synchronizing it to all of the systems that need it.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Data Engineering Weekly

JULY 7, 2024

Learn More → Notion: Building and scaling Notion’s data lake Notion writes about scaling the data lake by bringing critical data ingestion operations in-house. The dataset organization as a set of LSM trees seems an interesting approach, which sounds similar to Google’s Napa data warehouse system.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Zalando Engineering

OCTOBER 16, 2024

Introduction Context and Purpose Our team is part of the Transport teams within the Logistics department, where we build and manage software for internal users, including finance teams, warehouses, and, in the future, our third-party partners. Modularity was key to enabling independent feature development and deployment across teams.

Building 68

Building Transportation Coding Designing 68

Data Engineering Weekly

JULY 14, 2024

The blog highlights the reasoning behind selecting dbt and Dagster and some of the key improvements while adopting them, such as handling race conditions in dbt incremental update and bulk backfilling with Dagster. Metabase writes about why they build a continuous enrichment pipeline for customer contacts pipeline and its system design.

Data Engineering 98

Data Engineering Data Engineer Engineering Unstructured Data 98

Knowledge Hut

JUNE 26, 2023

This article is mainly focused on explaining different backend projects for beginners or students, intermediate learners, or those who have mid enough software development experience building large scalable projects. can be helpful for building a few awesome backend projects. What are Backend Development Projects?

Portfolio 98

Portfolio Project Food Media 98

Snowflake

APRIL 24, 2024

Building top-tier enterprise-grade intelligence using LLMs has traditionally been prohibitively expensive and resource-hungry, and often costs tens to hundreds of millions of dollars. Enterprises want to use LLMs to build conversational SQL data copilots, code copilots and RAG chatbots.

Amazon Web Services 145

Amazon Web Services SQL AWS Architecture 145

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data Architecture Data Pipeline 147

Data Engineering Weekly

AUGUST 6, 2024

CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems. However, managing data consistency across microservices can be challenging.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. In this blog, we will discuss: What is the Open Table format (OTF)? These systems are built on open standards and offer immense analytical and transactional processing flexibility.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Maxime Beauchemin

APRIL 26, 2023

This blog post is about prompt engineering, its challenges, and the advantages of using Promptimize to support engineers building prompts. Prompt Engineering In contrast, prompt engineering involves a more systematic and technical approach to designing and building systems, applications, or products that leverage AI models like GPT.

SQL 148

SQL Database Engineering Software Engineer 148

Netflix Tech

JUNE 14, 2023

This blog post will share broadly-applicable techniques (beyond GraphQL) we used to perform this migration. The AB experiment results hinted that GraphQL’s correctness was not up to par with the legacy system. We reached for a Netflix tool called the Sticky Canary to build confidence.

Utilities 143

Utilities Java Systems AWS 143

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. A variety of platforms have been developed to capture and analyze that information to great effect, but they are inherently limited in their utility due to their nature as storage systems.

Metadata 130

Metadata MongoDB MySQL Scala 130