Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake Building High Quality Data AWS 147

ProjectPro

JUNE 6, 2025

Microsoft offers Azure Data Lake, a cloud-based data storage and analytics solution. It is capable of effectively handling enormous amounts of structured and unstructured data. Therefore, it is a popular choice for organizations that need to process and analyze big data files.

Data Lake 40

Data Lake Amazon Web Services Data Warehouse SQL 40

ProjectPro

JUNE 6, 2025

No, that is not the only job in the data world. Data professionals who work with raw data, like data engineers, data analysts, machine learning scientists , and machine learning engineers , also play a crucial role in any data science project. Build your Data Engineer Portfolio with ProjectPro!

Data Engineer 59

Data Engineer Data Engineering Project Engineering 59

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

ProjectPro

JUNE 6, 2025

In the thought process of making a career transition from ETL developer to data engineer job roles? Read this blog to know how various data-specific roles, such as data engineer, data scientist, etc., Therefore, the need for data engineers is overgrowing. Is ETL required for data engineer?

Data Engineer 59

Data Engineer Data Engineering Engineering ETL Tools 59

ProjectPro

JUNE 6, 2025

If you are planning to make a career transition into data engineering and want to know how to become a data engineer, this is the perfect place to begin your journey. Beginners will especially find it helpful if they want to know how to become a data engineer from scratch. in the following few sections. .”

Data Engineer 58

Data Engineer Data Engineering Engineering Amazon Web Services 58

ProjectPro

JUNE 6, 2025

Data engineering is the foundation for data science and analytics by integrating in-depth knowledge of data technology, reliable data governance and security, and a solid grasp of data processing. Data engineers need to meet various requirements to build data pipelines.

AWS 52

AWS Data Engineer Data Engineering Engineering 52

ProjectPro

JUNE 6, 2025

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 40

Data Lake Data Warehouse Cloud Hadoop 40

KDnuggets

JANUARY 16, 2023

In this article, we will discuss the benefits of using SQL with a data lake and how it can help organizations unlock the full potential of their data.

Data Lake 108

Data Lake SQL Data IT 108

ProjectPro

JUNE 6, 2025

In recent years, you must have seen a significant rise in businesses deploying data engineering projects on cloud platforms. These businesses need data engineers who can use technologies for handling data quickly and effectively since they have to manage potentially profitable real-time data.

Data Engineer 40

Data Engineer Data Engineering Engineering Google Cloud 40

ProjectPro

JUNE 6, 2025

Microsoft's Azure Synapse Analytics (formerly SQL Data Warehouse) is a cloud data warehouse that combines data integration , data exploration, enterprise data warehousing, and big data analytics to offer a unified workspace for creating end-to-end analytics solutions.

Data Engineer 40

Data Engineer Data Engineering Engineering Data Lake 40

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Kafka 130

ProjectPro

JUNE 6, 2025

Due to this, knowledge of cloud computing platforms and tools is now essential for data engineers working with big data. Depending on the demands for data storage, businesses can use internal, public, or hybrid cloud infrastructure, including AWS , Azure , GCP , and other popular cloud computing platforms.

Data Engineer 40

Data Engineer Data Engineering PostgreSQL Engineering 40

ProjectPro

JUNE 6, 2025

Becoming a data engineer can be challenging, but we are here to make the journey easier. In this blog, we have curated a list of the best data engineering courses so you can master this challenging field with confidence. Say goodbye to confusion and hello to a clear path to data engineering expertise!

Data Engineer 45

Data Engineer Data Engineering Cloud Engineering 45

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

ProjectPro

JUNE 6, 2025

This blog is your one-stop solution for the top 100+ Data Engineer Interview Questions and Answers. In this blog, we have collated the frequently asked data engineer interview questions based on tools and technologies that are highly useful for a data engineer in the Big Data industry.

Data Engineer 40

Data Engineer Data Engineering Engineering Hadoop 40

ProjectPro

JUNE 6, 2025

This blog will help you understand what data engineering is with an exciting data engineering example, why data engineering is becoming the sexier job of the 21st century is, what is data engineering role, and what data engineering skills you need to excel in the industry, Table of Contents What is Data Engineering?

Data Science 40

Data Science Data Engineer Data Engineering Engineering 40

ProjectPro

JUNE 6, 2025

Azure Databricks embodies this philosophy by providing a user-friendly interface that simplifies data engineering complexities, helping professionals extract meaningful insights and drive business value. According to a report by IDC, worldwide data generation is projected to reach a staggering 175 zettabytes by 2025.

Data Engineer 40

Data Engineer Data Engineering Engineering Data Lake 40

ProjectPro

JUNE 6, 2025

This blog post provides an overview of the top 10 data engineering tools for building a robust data architecture to support smooth business operations. Table of Contents What are Data Engineering Tools? Dice Tech Jobs report 2020 indicates Data Engineering is one of the highest in-demand jobs worldwide.

Data Engineer 45

Data Engineer Data Engineering Engineering Kafka 45

ProjectPro

JUNE 6, 2025

Previously, the spotlight was on gaining relevant insights from data, but recently, data handling has gained attention. Because of that, data engineer jobs have garnered recognition and popularity. Most of us must have used Google Drive to share data among peers at least once in a lifetime.

Data Engineer 49

Data Engineer Data Engineering Engineering Hadoop 49

ProjectPro

JUNE 6, 2025

Businesses are finding new methods to benefit from data. Data engineering entails building data pipelines for ingesting, modifying, supplying, and sharing data for analysis. Therefore, every decision is reviewed with this approach to develop new data-driven judgments. What is ELT?

Data Engineer 40

Data Engineer Data Engineering Engineering Data Lake 40

ProjectPro

JUNE 6, 2025

Becoming a successful aws data engineer demands you to learn AWS for data engineering and leverage its various services for building efficient business applications. AWS has become one of the prime choices of cloud platforms for anyone who wants to learn about dealing with data at scale! What is Data Engineering??

AWS 40

AWS Data Engineer Data Engineering Engineering 40

ProjectPro

JUNE 6, 2025

Azure Data Factory is a popular tool that orchestrates data flow and transformation between multiple data repositories and resources. Table of Contents What is Azure Data Factory? Why do data engineers love Azure Data Factory? Data Control : Invoke other pipelines, Run SSIS packages, etc.

Data Engineer 40

Data Engineer Data Engineering Engineering Hospitality 40

ProjectPro

JUNE 6, 2025

Its intuitive, and data engineer-friendly interface helps anyone efficiently work with data at scale. The No-Code orchestration offered by Data Factory makes it an effective tool for any data engineer. The demand for data engineering will only grow as the data industry grows.

Data Lake 60

Data Lake Metadata SQL Datasets 60

ProjectPro

JUNE 6, 2025

The ADF service makes it easy to plan and automate data-driven processes(data pipelines) for data engineering projects that can consume data from multiple sources. You can easily use these custom logs to conduct SQL queries on your meta-store and assess your data quality.

Data Engineer 40

Data Engineer Data Engineering Project Engineering 40

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Snowflake

NOVEMBER 26, 2024

Before it migrated to Snowflake in 2022, WHOOP was using a catalog of tools — Amazon Redshift for SQL queries and BI tooling, Dremio for a data lake, PostgreSQL databases and others — that had ultimately become expensive to manage and difficult to maintain, let alone scale.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Data Lake 105

ProjectPro

JUNE 6, 2025

Azure Data Factory 2. Azure Data Lake Storage 7. Azure Logic Apps Azure ETL Best Practices for Big Data Projects Get Your Hands-on Azure ETL Projects with ProjectPro! He explores their collaborative potential in orchestrating, exploring, and analyzing data, shaping a secure and comprehensive data engineering landscape.

ETL Tools 40

ETL Tools Data Engineer Data Engineering Data Lake 40

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 115

Data Lake Data Warehouse Cloud Unstructured Data 115

ProjectPro

JUNE 6, 2025

With over 175 full features service offerings, organizations are head hunting for AWS data engineers who can help them build and maintain the entire AWS cloud infrastructure to keep the applications up and running. Cloud platforms are becoming the new standard for managing an organization's data.

AWS 45

AWS Data Engineer Data Engineering Amazon Web Services 45

ProjectPro

JUNE 6, 2025

Welcome to our guide on How to Crack the Amazon Data Engineer Interview in 2024! million, Amazon heavily relies on data engineers for its success. With a 30% year-over-year increase in hiring data engineers, Amazon underscores its commitment to leveraging big data effectively.

Data Engineer 40

Data Engineer Data Engineering Engineering NoSQL 40

ProjectPro

JUNE 6, 2025

Experts predict that by 2025, the global big data and data engineering market will reach $125.89 With the right tools, mindset, and hands-on experience, you can become a key player in transforming how organizations use data to drive innovation and decision-making. But what does it take to become an ETL Data Engineer?

Data Engineer 40

Data Engineer Data Engineering Engineering ETL Tools 40

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineer Data Engineering Cloud 130

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake 182

Data Lake High Quality Data Data Pipeline Government 182

Cloudera

MARCH 2, 2021

However, in the typical enterprise, only a small team has the core skills needed to gain access and create value from streams of data. This data engineering skillset typically consists of Java or Scala programming skills mated with deep DevOps acumen. SQL as the democratization enabler. A rare breed.

SQL 112

SQL Data Lake Java Scala 112

Snowflake

NOVEMBER 11, 2024

In addition to free assessments and free table conversions, SnowConvert now supports accurate conversion of database views from Teradata, Oracle or SQL Server for free. Sensitive data can have enormous value but is oftentimes locked down due to privacy requirements.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105