Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Are data agents harder to build?

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Knowledge Hut

OCTOBER 27, 2023

System design projects involve creating a plan for a complex system that will meet specific requirements or solve a particular problem. These projects may involve designing software applications, network infrastructures, manufacturing processes, or other complex systems. What is System Design?

Designing 52

Designing Systems Project Cloud Computing 52

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.

Systems 93

Systems Media Machine Learning Data Warehouse 93

Data Engineering Podcast

JULY 27, 2020

Summary A majority of the scalable data processing platforms that we rely on are built as distributed systems. Kyle Kingsbury created the Jepsen framework for testing the guarantees of distributed data processing systems and identifying when and why they break. Can you start by describing what the Jepsen project is?

Systems 100

Systems Building Scala Java 100

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

MARCH 31, 2024

Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. Dagster offers a new approach to building and running data platforms and data pipelines.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

MAY 5, 2024

In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. However, there are some problems with step (4): The build is not properly isolated: it can (in theory) fetch arbitrary things from the network, access arbitrary files on the filesystem, and even modify other packages.

Project 145

Project Programming Language Building Systems 145

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Data Engineering Podcast

JANUARY 21, 2019

Summary Building internal expertise around big data in a large organization is a major competitive advantage. They discuss the challenges of being at global scale from the start, hiring and training talented engineers, prototyping and deploying new systems in the cloud, and what they have learned in the process.

Big Data 100

Big Data Systems Building Hadoop 100

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. If you've learned something or tried out a project from the show then tell us about it! Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. In this episode he shares the design of the project and how it fits into your development practices. What are the different places in a data system that schema definitions need to be established?

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. What are the core problems that you were addressing with this project? We feel your pain.

Building 147

Building Machine Learning Metadata Data Integration 147

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Decodable was built with a mission of eliminating all of the painful aspects of developing and deploying stream processing systems for engineering teams. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Knowledge Hut

FEBRUARY 1, 2024

Not only could this recommendation system save time browsing through lists of movies, it can also give more personalized results so users don’t feel overwhelmed by too many options. What are Movie Recommendation Systems? Recommender systems have two main categories: content-based & collaborative filtering.

Systems 98

Systems Entertainment Algorithm Datasets 98

Knowledge Hut

OCTOBER 23, 2023

Both the project leader and project manager roles are crucial to a project's success if project management is your area of interest as a career. Research and introspection are required to comprehend and decide which role is best for you, especially if you are interested in pursuing a career in project management.

Project 98

Project Management Generalist Algorithm 98

Rockset

MARCH 28, 2024

Overview In this guide, you will: Gain a high-level understanding of vectors, embeddings, vector search, and vector databases, which will clarify the concepts we will build upon. Build a dynamic web application using vanilla CSS, HTML, JavaScript, and Flask, seamlessly integrating with the Rockset API and the OpenAI API.

Systems 52

Systems Building Database Utilities 52

Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. RudderStack helps you build a customer data platform on your warehouse or data lake. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Knowledge Hut

APRIL 16, 2024

Over several decades, projects have been initiated or undertaken due to market demands, business needs, at the behest of customer request, technological advancements and to comply with regulatory requirements. We will look at some of the key project management methods followed in today’s world. Know more about the project description.

Project 98

Project Management Programming Designing 98

Knowledge Hut

MAY 6, 2024

The scope keeps changing as the project makes progress on the agreed timeline, so, the estimation becomes a challenging task for the product owner. Agile software estimation is used for realistic planning to implementing the project requirements as per commitments. Master the art of project management and achieve new heights.

Project 98

Project Programming Certification Database 98

Knowledge Hut

JUNE 26, 2023

Having knowledge of real-world software applications or projects are very essential for any projects for backend developers aspiring software engineers or developers. The portfolio projects showcase their talents and skills whenever they try to look for new opportunities and jobs. What are Backend Development Projects?

Portfolio 98

Portfolio Project Food Media 98

Data Engineering Podcast

JANUARY 28, 2024

Summary Monitoring and auditing IT systems for security events requires the ability to quickly analyze massive volumes of unstructured log data. If you've learned something or tried out a project from the show then tell us about it! The Machine Learning Podcast helps you go from idea to production with machine learning.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Knowledge Hut

MAY 6, 2024

Applying systems thinking views a system as a set of interconnected and interdependent components defined by its limits and more than the sum of their parts (subsystems). When one component of a system is altered, the effects frequently spread across the entire system. are the main objectives of systems thinking.

Systems 98

Systems Transportation Certification Healthcare 98

Knowledge Hut

OCTOBER 27, 2023

In today's fast-paced technological environment, software engineers are continually seeking innovative projects to hone their skills and stay ahead of industry trends. Engaging in software engineering projects not only helps sharpen your programming abilities but also enhances your professional portfolio. cvtColor(image, cv2.COLOR_BGR2GRAY)

Software Engineer 98

Software Engineer Software Engineering Coding Project 98

Knowledge Hut

OCTOBER 29, 2023

Choosing the best computer science project topic is critical to the success of any computer science student or employee. After all, the more engaging and interesting topic, the more likely it is that students or employees will be able to stay motivated and focused throughout the duration of the project.

Computer Science 111

Computer Science Coding Project Hospitality 111

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. These systems are built on open standards and offer immense analytical and transactional processing flexibility. 2019 - Delta Lake Databricks released Delta Lake as an open-source project.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Podcast

DECEMBER 2, 2018

Summary Distributed systems are complex to build and operate, and there are certain primitives that are common to a majority of them. Rather then re-implement the same capabilities every time, many projects build on top of Apache Zookeeper. Can you start by explaining what Zookeeper is and how the project got started?

Systems 100

Systems Building Kafka Java 100

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode Ian Schweer shares his experiences at Riot Games supporting player-focused features such as machine learning models and recommeder systems that are deployed as part of the game binary. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

OCTOBER 23, 2022

Summary Agile methodologies have been adopted by a majority of teams for building software applications. Applying those same practices to data can prove challenging due to the number of systems that need to be included to implement a complete feature. When is agile the wrong choice for a data project?

Project 130

Project Metadata MongoDB MySQL 130

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Why Postgres?

Database 130

Database Building PostgreSQL BI 130

Knowledge Hut

APRIL 12, 2024

Project management plays a significant role in the success of every organization. It ensures that the project is on track, aids in efficient management of resources, and also keeps the stakeholders know what is project and what's happening in it. What is Project Management Structures, and Why is it Important?

Project 98

Project Management Certification Designing 98

Precisely

SEPTEMBER 26, 2024

Building data literacy across your organization empowers teams to make better use of AI tools. However, achieving success in AI projects isn’t just about deploying advanced algorithms or machine learning models. The real challenge lies in ensuring that the data powering your projects is AI-ready.

Project 75

Project Government Data Governance Data Integration 75

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. What technical systems are required to generate and collect those interactions?

Building 130

Building IT Metadata MongoDB 130

Engineering at Meta

DECEMBER 6, 2023

This is the most significant milestone yet for this project, which began in earnest after Mark Zuckerberg outlined his vision for it in 2019. Throughout the project, we have consulted with a diverse range of external parties to ensure that we’re making the right set of tradeoffs.

Building 144

Building Designing Consulting Coding 144

Zalando Engineering

OCTOBER 16, 2024

Introduction Context and Purpose Our team is part of the Transport teams within the Logistics department, where we build and manage software for internal users, including finance teams, warehouses, and, in the future, our third-party partners. This flexibility played a crucial role in the architecture of our portal project.

Building 68

Building Transportation Coding Designing 68

Tweag

JULY 26, 2023

The vast majority of the Rust projects are using Cargo as a build tool. Cargo is great when you are developing and packaging a single Rust library or application, but when it comes to a fast-growing and complex workspace, one could be attracted to the idea of using a more flexible and scalable build system.

Building 97

Building Metadata Coding Project 97