Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Data Engineering Podcast

DECEMBER 3, 2023

Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Systems 130

Systems Designing Data Lake SQL 130

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 363

Education Project PostgreSQL Software Engineer 363

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. What are the technical systems that you are relying on to power the different data domains?

Building 147

Building Machine Learning Metadata Data Integration 147

Data Engineering Podcast

MAY 5, 2024

In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building 147

Building Data Lake High Quality Data Machine Learning 147

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.    He then worked at the casual games company Zynga, building their in-game advertising platform. Subscribe here.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Knowledge Hut

FEBRUARY 1, 2024

Machine learning (ML) offers just that - an advanced technology to help users find the exact films they want. Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems?

Systems 98

Systems Entertainment Algorithm Datasets 98

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When is Scanner the wrong choice?

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. What are the pain points that are still prevalent in lakehouse architectures as compared to warehouse or vertically integrated systems?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Engineering at Meta

DECEMBER 6, 2023

For example, we recently published an updated white paper, “ Meta’s Approach to Safer Private Messaging on Messenger and Instagram Direct Messaging ,” t hat sets out the industry-leading safety systems and tools available on Messenger. End-to-end encryption isn’t about the technology at its core.

Building 145

Building Designing Consulting Accessible 145

Data Engineering Podcast

APRIL 16, 2023

As the capabilities of these systems has improved and become more accessible, the target of what self-serve means changes. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

JANUARY 15, 2023

Summary The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. In this episode Srujan Akula explains how the system is implemented and how you can start using it today with your existing data systems.

Coding 130

Coding Building PostgreSQL Data Lake 130

Knowledge Hut

MARCH 22, 2024

I have comprehensively analyzed the area of physical security, particularly the ongoing discussion surrounding fail safe vs fail-safe secure electric strike locking systems. These two distinct approaches offer unique strategies for securing buildings and valuables. What are Fail-Safe Door Locks?

Systems 105

Systems Electronics Hospitality Architecture 105

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Why Postgres? Closing Announcements Thank you for listening!

Database 130

Database Building PostgreSQL BI 130

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. In this episode Gleb Mezhanskiy shares some valuable advice and insights into how you can build reliable and well-tested data assets with dbt and data-diff.

Project 130

Project Building Data Lake Machine Learning 130

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Data Engineering Podcast

JANUARY 29, 2023

The team at Omni have taken a new approach by automatically building models based on the queries that are executed. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems. What are the core goals that you are trying to achieve with building Omni?

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Cloudera

DECEMBER 1, 2023

Recent Government Initiatives on Public Sector AI Solutions In recent years, governments across the globe have recognized the transformative potential of artificial intelligence (AI) and have embarked on initiatives to harness this technology to drive innovation and serve their citizens more effectively. million), among others.

Building 115

Building Government Transportation Data Governance 115

Engineering at Meta

DECEMBER 10, 2024

Were explaining the end-to-end systems the Facebook app leverages to deliver relevant content to people. At Facebooks scale, the systems built to support and overcome these challenges require extensive trade-off analyses, focused optimizations, and architecture built to allow our engineers to push for the same user and business outcomes.

Systems 74

Systems Architecture Engineering Data Pipeline 74

The Pragmatic Engineer

OCTOBER 10, 2023

He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. I wrote code for drivers on Windows, and started to put a basic observability system in place.

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). Technological innovation rarely happens in a vacuum; it builds on previous technologies.

Programming Language 255

Programming Language Project Coding Engineering 255

The Pragmatic Engineer

MAY 11, 2023

A quick summary of these technologies: Prometheus : a time series database. A very popular open-source solution for systems and services monitoring. A fast and open-source column-oriented database management system, which is a popular choice for log management. It evaluates rules and can trigger alerts. But why is this?

AWS 318

AWS Software Engineer Software Engineering Google Cloud 318

Engineering at Meta

NOVEMBER 15, 2023

Meta is building for the future of AI at every level – from hardware like MTIA v1 , Meta’s first-generation AI inference accelerator to publicly released models like Llama 2 , Meta’s next-generation large language model, as well as new generative AI (GenAI) tools like Code Llama.

Building 103

Building Engineering Transportation Software Engineer 103

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. In this episode Nick Schrock, creator of Dagster, shares his perspective on the state of data orchestration technology and its application to help inform its implementation in your environment.

BI 208

BI SQL Data Machine Learning 208

Data Engineering Podcast

APRIL 21, 2024

Summary Generative AI has rapidly transformed everything in the technology sector. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Cloudera

NOVEMBER 1, 2024

It also provided a moment for us to launch an important initiative for Cloudera: our Women Leaders in Technology (WLIT) initiative. Building Inclusive Data-Driven Organizations: Leadership Strategies for the Modern Workplace As it stands, women currently account for approximately 25% of the technology workforce.

Technology 97

Technology Recruitment Banking AWS 97

The Pragmatic Engineer

AUGUST 1, 2023

Senior Engineers are not only expected to lead significant projects in their teams, but they have a say in whether that feature is worth building or not. They strive to build things fast, but also know when we have to slow down to go faster.” Address systemic issues. Reduce support costs by addressing systemic issues.

Software Engineer 245

Software Engineer Software Engineering Engineering Designing 245

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. Dagster offers a new approach to building and running data platforms and data pipelines. There are numerous stream processing engines, near-real-time database engines, streaming SQL systems, etc.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

MARCH 3, 2024

Summary Artificial intelligence technologies promise to revolutionize business and produce new sources of value. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Your first 30 days are free!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” And COBOL was just one of many attempts.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

Data Engineering Podcast

OCTOBER 29, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. When a team is selecting a database technology for a given task, what are the types of questions that they should be asking?

Database 189

Database BI SQL Machine Learning 189