Data Engineering Podcast

JUNE 4, 2023

Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Lake 130

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. You’re left wondering if there’s a breaking point where your DIY data solution won’t cut it anymore—and honestly, you might be there already. It means you’re scaling!

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Introduction 2. Objective 3. Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6.

Data Engineer 246

Data Engineer Data Engineering Project Data Lake 246

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Weekly

SEPTEMBER 15, 2024

Uber: DataMesh - How Uber laid the foundations for the Data Lake Cloud migration Many companies are slowly adopting DataMesh, and Uber writes about adopting the data mesh principle. Whether or not Data Mesh is a separate product is debatable, but it is certainly an impactful framework for scaling data platforms.

Data Engineer 67

Data Engineer Data Engineering Engineering Data Lake 67

Data Engineering Weekly

NOVEMBER 11, 2024

Slack: Empowering Engineers with AI In our most recent internal developer survey, 65% of respondents said they prefer an IDE as the primary platform to engage with generative-AI for development tasks. link] Yelp: Loading data into Redshift with DBT It is good to see a blog about Redshift unrelated to migrating.

Data Engineer 57

Data Engineer Data Engineering Engineering Datasets 57

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Data Engineering Weekly

JULY 7, 2024

Learn More → Notion: Building and scaling Notion’s data lake Notion writes about scaling the data lake by bringing critical data ingestion operations in-house. Hudi seems to be a de facto choice for CDC data lake features. Notion migrated the insert heavy workload from Snowflake to Hudi.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Knowledge Hut

JULY 5, 2024

Large enterprises heavily rely on data for informed decision-making, and this reliance is where data engineers step in. Data engineers like myself play a pivotal role in assessing infrastructure and taking relevant actions. Looking ahead, the future of data engineering appears promising.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Cleanse 52

Analytics Vidhya

FEBRUARY 2, 2023

Now, businesses are looking for different types of data storage to store and manage their data effectively. Organizations can collect millions of data, but if they’re lacking in storing that data, those efforts […] The post A Comprehensive Guide to Data Lake vs. Data Warehouse appeared first on Analytics Vidhya.

Data Lake 202

Data Lake Data Warehouse Data Storage Data 202

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineer Data Engineering Cloud 130

Towards Data Science

DECEMBER 4, 2023

A Glossary with Use Cases for First-Timers in Data Engineering An happy Data Engineer at work Are you a data engineering rookie interested in knowing more about modern data infrastructures? In this guide Data Engineering meets Formula 1. I bet you are, this article is for you!

Data Engineer 81

Data Engineer Data Engineering Engineering Data Lake 81

Analytics Vidhya

FEBRUARY 25, 2023

Introduction A data lake is a centralized and scalable repository storing structured and unstructured data. The need for a data lake arises from the growing volume, variety, and velocity of data companies need to manage and analyze.

Cloud Storage 157

Cloud Storage Data Lake Cloud Unstructured Data 157

Data Engineering Weekly

APRIL 21, 2024

link] Stripe: Shepherd - How Stripe adapted Chronon to scale ML feature development Airbnb recently open-sourced Chronon , a declarative feature engineering framework. link] Canva: Scaling to Count Billions If you know how to count, you’re an excellent data engineer. Counting is the hardest problem in data engineering.

Data Engineer 52

Data Engineer Data Engineering Engineering Medical 52

Data Engineering Weekly

MARCH 3, 2024

Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering. Part 1: Why did we need to build our own SIEM?

Data Engineer 67

Data Engineer Data Engineering Engineering Pipeline-centric 67

Data Engineering Podcast

MAY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex. My thanks to the team at Code Comments for their support.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Multiple open source projects and vendors have been working together to make this vision a reality.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

NOVEMBER 12, 2023

Generative AI has accelerated the ability of developer tools to provide useful suggestions that speed up the work of engineers. Tabnine is one of the main platforms offering an AI powered assistant for software engineers. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Data Engineering Weekly

FEBRUARY 4, 2024

Joe Reis: Definition of Data Modeling & What Data Modeling Is not Joe raised a very fundamental question in data engineering. What is Data Modeling, and what is not? He rightly points out that many data engineers will name any of the modeling techniques as data modeling.

Data Engineer 62

Data Engineer Data Engineering Engineering PostgreSQL 62

Data Engineering Weekly

MAY 12, 2024

We have a special offer for Data Engineering Weekly readers: start a free trial and get $500 credits for free, so you can see just how easy it is to spin up your own analytics with us! Atlassian emphasizes how crucial it is for its data lake, utilized by over half of its employees, to support continuous growth and adaptability.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Lake 52

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Data Engineering Weekly

APRIL 28, 2024

Intuit: The Data Mesh Strategy Behind Intuit’s Global Financial Technology Platform The Data Product Builder platform is becoming increasingly important in enterprise data engineering. It offers more targeted and customized data asset building than the general-purpose data stack.

Data Engineer 52

Data Engineer Data Engineering Engineering Hospitality 52

Data Engineering Weekly

AUGUST 11, 2024

link] Uber: Enabling Security for Hadoop Data Lake on Google Cloud Storage Uber writes about securing a Hadoop-based data lake on Google Cloud Platform (GCP) by replacing HDFS with Google Cloud Storage (GCS) while maintaining existing security models like Kerberos-based authentication.

Data Engineer 57

Data Engineer Data Engineering Google Cloud Engineering 57

Data Engineering Podcast

JUNE 23, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Data lakes in various forms have been gaining significant popularity as a unified interface to an organization's analytics. When is Fabric the wrong choice?

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

Data Engineering Weekly

MAY 5, 2024

[link] Uber: From Predictive to Generative – How Michelangelo Accelerates Uber’s AI Journey Constantly adopting and implementing tech advancement with an existing system indicates efficient engineering. Uber wrote an in-depth article about the evolution of its centralized ML platform, Michelangelo.

Data Engineer 67

Data Engineer Data Engineering Engineering Google Cloud 67

Data Engineering Podcast

SEPTEMBER 19, 2021

In this episode Jillian Rowe shares her experience of working in the field and supporting teams of scientists and analysts with the data infrastructure that they need to get their work done. This is a fascinating exploration of the collaboration between data professionals and scientists. Missing data? Stale dashboards?

Data Engineer 100

Data Engineer Data Engineering Engineering Data Lake 100

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Weekly

MARCH 24, 2024

Dive into Spyne's experience with: - Their search for query acceleration with pre-aggregations and caching - Developing new functionality with Open AI - Optimizing query cost with their data warehouse [link] Suresh Hasuni: Cost Optimization Strategies for Scalable Data Lakehouse Cost is the major concern as the adoption of data lakes increases.

Data Engineer 62

Data Engineer Data Engineering Engineering Metadata 62

Data Engineering Weekly

DECEMBER 25, 2023

Welcome to another insightful edition of Data Engineering Weekly. As we approach the end of 2023, it's an opportune time to reflect on the key trends and developments that have shaped the field of data engineering this year. In conclusion, 2023 has been a year of significant developments and shifts in data engineering.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Seattle Data Guy

DECEMBER 19, 2022

Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineer 52

Data Engineer Data Engineering Python Engineering 52

Data Engineering Weekly

DECEMBER 4, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. link] Barr Moses: What’s Next for Data Engineering in 2023?

Data Engineer 52

Data Engineer Data Engineering Engineering Data Lake 52

Data Engineering Podcast

JULY 17, 2022

Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. RudderStack helps you build a customer data platform on your warehouse or data lake. Data teams are increasingly under pressure to deliver.

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100