Data Engineering Podcast

JUNE 17, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How has the move to the cloud for data warehousing/data platforms influenced the practice of data modeling?

Data Lake 130

Data Lake Metadata Machine Learning Data Architecture 130

ProjectPro

JUNE 6, 2025

This beginner's guide will give you a detailed overview of Azure Synapse Analytics and its architecture to help you build enterprise-grade data pipelines for your next data analytics project. Why Use Azure Synapse Analytics For Big Data Analytics Projects? Well, Azure Synapse Analytics is your answer!

Data Engineering 40

Data Engineering Data Engineer Engineering Data Lake 40

Data Engineering Weekly

FEBRUARY 23, 2025

[link] Alireza Sadeghi: Open Source Data Engineering Landscape 2025 This article comprehensively overviews the 2025 open-source data engineering landscape, highlighting key trends, active projects, and emerging technologies.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

ProjectPro

JUNE 6, 2025

As we can see, it turns out that the data engineering role requires a vast knowledge of different big data tools and technologies. The data engineering role requires professionals who can build various data pipelines to enable data-driven models. Maintain the data architecture over time and its scalability.

Data Science 40

Data Science Data Engineering Data Engineer Engineering 40

ProjectPro

JUNE 6, 2025

Explore what is Apache Iceberg, what makes it different, and why it’s quickly becoming the new standard for data lake analytics. Data lakes were born from a vision to democratize data, enabling more people, tools, and applications to access a wider range of data. Apache Iceberg Architecture 1.

Architecture 40

Architecture Data Lake Metadata Cloud Storage 40

ProjectPro

JUNE 6, 2025

Busting the Myth of Zero ETL: Why is Data Transformation so critical? Zero ETL Components Zero ETL Benefits Zero ETL Use Cases AWS Zero ETL Integrations Learn Building Scalable Zero ETL Data Pipelines with ProjectPro! Data Lake Architecture: Data lakes store raw, unstructured data, making it instantly available for analysis.

Data Analytics 40

Data Analytics MySQL PostgreSQL Data Lake 40

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

ProjectPro

JUNE 6, 2025

Here are four key benefits of pursuing these courses- Enhanced Cloud Data Management Skills Data engineering courses provide cloud professionals or data engineers with the knowledge and skills to effectively manage data in cloud environments. Ratings/Reviews This certificate program has an overall rating of 4.6

Data Engineering 45

Data Engineering Data Engineer Cloud Engineering 45

Data Engineering Weekly

SEPTEMBER 18, 2024

The Rise of Data Observability Data observability has become increasingly critical as companies seek greater visibility into their data processes. This growing demand has found a natural synergy with the rise of the data lake. As a result, monitoring data in real time was often an afterthought.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Data 98

ProjectPro

JUNE 6, 2025

It also offers a unique architecture that allows users to quickly build tables and begin querying data without administrative or DBA involvement. Snowflake is a cloud-based data platform that provides excellent manageability regarding data warehousing, data lakes, data analytics, etc.

Architecture 56

Architecture IT Data Warehouse Amazon Web Services 56

ProjectPro

JUNE 6, 2025

This blog post provides an overview of the top 10 data engineering tools for building a robust data architecture to support smooth business operations. Table of Contents What are Data Engineering Tools? Data engineers manage that massive amount of data using various data engineering tools, frameworks, and technologies.

Data Engineering 45

Data Engineering Data Engineer Engineering Kafka 45

Monte Carlo

JANUARY 23, 2023

When it comes to the data community, there’s always a debate broiling about something— and right now “data mesh vs data lake” is right at the top of that list. In this post we compare and contrast the data mesh vs data lake to illustrate the benefits of each and help discover what’s right for your data platform.

Data Lake 52

Data Lake Business Intelligence Government Unstructured Data 52

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind. Data discovery tools and platforms can help.

Data Lake 52

Data Lake Data Warehouse Government Unstructured Data 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka Data Lake 52

ProjectPro

JUNE 6, 2025

Key Responsibilities of a Data Engineer Here are the skills to hone for fulfilling the day-to-day responsibilities of a data engineer: Obtain data from third-party providers with the help of robust API integrations. Build, Design, and maintain data architectures using a systematic approach that satisfies business needs.

Data Engineering 49

Data Engineering Data Engineer Engineering Hadoop 49

DataKitchen

NOVEMBER 29, 2021

Data Gets Meshier. 2022 will bring further momentum behind modular enterprise architectures like data mesh. The data mesh addresses the problems characteristic of large, complex, monolithic data architectures by dividing the system into discrete domains managed by smaller, cross-functional teams.

Manufacturing 98

Manufacturing Data Lake Architecture Data Architecture 98

ProjectPro

JUNE 6, 2025

It is tailored for professionals aspiring to demonstrate proficiency in leveraging AWS services to extract, transform, and load (ETL) data for analysis. It also emphasizes skills in optimizing data storage, ensuring data quality, and implementing scalable and secure data solutions on the AWS platform. PREVIOUS NEXT <

AWS 40

AWS Certification Data Engineering Data Engineer 40

ProjectPro

JUNE 6, 2025

Cloud platforms leverage various solutions to provide users with better insights, including Data Migration , Data Engineering, and Data Analytics. AWS Data Engineering is one of the core elements of AWS Cloud in delivering the ultimate solution to users. Table of Contents Who is an AWS Data Engineer?

AWS 45

AWS Data Engineering Data Engineer Amazon Web Services 45

Data Engineering Podcast

NOVEMBER 18, 2019

Summary With the constant evolution of technology for data management it can seem impossible to make an informed decision about whether to build a data warehouse, or a data lake, or just leave your data wherever it currently rests. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo!

Data Lake 100

Data Lake Scala Data Warehouse Programming Language 100

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

ProjectPro

JUNE 6, 2025

It is easy to query structured data and perform further analysis on it. It is difficult to query the required unstructured data. Relational databases and data warehouses contain structured data. Data lakes and non-relational databases can contain unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

ProjectPro

JUNE 6, 2025

Real-time monitoring and logging of ETL processes provide insights into the health of the data pipeline, facilitating the timely identification of data quality issues. Regular data auditing and reconciliation practices help detect and rectify discrepancies, ensuring consistency throughout the process.

Data Engineering 40

Data Engineering Data Engineer Engineering NoSQL 40

DataKitchen

NOVEMBER 4, 2021

Data organizations often have a mix of centralized and decentralized activity. DataOps concerns itself with the complex flow of data across teams, data centers and organizational boundaries. It expands beyond tools and data architecture and views the data organization from the perspective of its processes and workflows.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Edureka

APRIL 9, 2025

Imagine being in charge of creating an intelligent data universe where collaboration, analytics, and artificial intelligence all work together harmoniously. Development of Some Relevant Skills and Knowledge Data Engineering Fundamentals: Theoretical knowledge of data loading patterns, data architectures, and orchestration processes.

Engineering 40

Engineering Data Ingestion Data Lake Programming Language 40

Cloudera

NOVEMBER 18, 2020

The technological linchpin of its digital transformation has been its Enterprise Data Architecture & Governance platform. It hosts over 150 big data analytics sandboxes across the region with over 200 users utilizing the sandbox for data discovery.

Medical 123

Medical Banking Telecommunication Government 123

Data Engineering Podcast

SEPTEMBER 18, 2019

They have built the “missing” Amazon Redshift console – it’s an amazing analytics product for data engineers to find and re-write slow queries and gives actionable recommendations to optimize data pipelines. WeWork, Postmates, and Medium are just a few of their customers.

Data Lake 100

Data Lake Hadoop BI Kafka 100

Data Engineering Podcast

OCTOBER 21, 2019

This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC. Raghu Murthy, founder and CEO of Datacoral built data infrastructures at Yahoo! This week’s episode is also sponsored by Datacoral, an AWS-native, serverless, data infrastructure that installs in your VPC.

Cloud 100

Cloud Data Lake Programming Language Hadoop 100

ProjectPro

JUNE 6, 2025

As businesses continue to recognize the value of efficient data management, the demand for certified data engineers has surged. These roles typically involve working with large-scale data solutions, implementing data pipelines, and optimizing data architectures for performance and scalability.

Data Engineering 40

Data Engineering Data Engineer Google Cloud Engineering 40

ProjectPro

JUNE 6, 2025

Microsoft offers an entry-level azure certification that validates your skills and knowledge of working with various Azure Data Services, including core concepts and technologies like Azure Data Lake , Azure Synapse Analytics, and Azure Data Factory. In this project, you will perform ETL on the Movielens dataset.

Certification 45

Certification Google Cloud Data Lake SQL 45

ProjectPro

JUNE 6, 2025

Azure Data Engineer Associate DP-203 Certification Candidates for this exam must possess a thorough understanding of SQL , Python, and Scala , among other data processing languages. Must be familiar with data architecture, data warehousing, parallel processing concepts, etc.

Certification 40

Certification Data Engineering Data Engineer Engineering 40

AltexSoft

OCTOBER 30, 2021

They’re integral specialists in data science projects and cooperate with data scientists by backing up their algorithms with solid data pipelines. Juxtaposing data scientist vs engineer tasks. One data scientist usually needs two or three data engineers. Providing data access tools.

Data Engineering 98

Data Engineering Data Engineer Engineering Machine Learning 98

Snowflake

JULY 22, 2024

Determining an architecture and a scalable data model to integrate more source systems in the future. The benefits of migrating to Snowflake start with its multi-cluster shared data architecture, which enables scalability and high performance. top modernizing your data lake with Snowflake, watch our on demand webinar.

Government 119

Government Cloud Data Data Lake 119

Cloudera

DECEMBER 1, 2022

Full-stack observability is a critical requirement for effective modern data platforms to deliver the agile, flexible, and cost-effective environment organizations are looking for. RI is a global leader in the design and deployment of large-scale, production-level modern data platforms for the world’s largest enterprises.

Data Lake 84

Data Lake Data Pipeline Data Governance Analytics Application 84

ProjectPro

JUNE 6, 2025

Building Data Pipelines in Azure Building data pipelines in Azure with Azure Synapse Analytics empowers data engineers to process and analyze vast amounts of data efficiently. Azure DP 203 Exam Pricing The pricing for this Microsoft Azure exam involves an exam fee and optional training costs.

Certification 40

Certification Data Storage Big Data Data Engineering 40

Cloudera

OCTOBER 7, 2021

Those decentralization efforts appeared under different monikers through time, e.g., data marts versus data warehousing implementations (a popular architectural debate in the era of structured data) then enterprise-wide data lakes versus smaller, typically BU-Specific, “data ponds”.

Architecture 124

Architecture Metadata Kafka Government 124