Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. Email hosts@dataengineeringpodcast.com ) with your story.

Data Process 147

Data Process Process Data Lake High Quality Data 147

ProjectPro

JUNE 6, 2025

Loading data into a DataFrame Here, you will explore different methods to load external data into a DataFrame. It covers reading data from CSV, Excel, JSON format file, and SQL databases. You will understand how to customize the import process, handle null values, and specify data types during data loading.

Data Process 40

Data Process Process Aggregated Data Data Science 40

Pinterest Engineering

NOVEMBER 18, 2024

Change Data Capture (CDC) is a crucial technology that enables organizations to efficiently track and capture changes in their databases. In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. What is Change Data Capture?

Kafka 54

Kafka MySQL Database Software Engineer 54

ProjectPro

JUNE 6, 2025

Looking for an efficient tool for streamlining and automating your data processing workflows? Let's consider an example of a data processing pipeline that involves ingesting data from various sources, cleaning it, and then performing analysis. Airflow operators hold the data processing logic.

Data Pipeline 63

Data Pipeline PostgreSQL Python Database 63

ProjectPro

JUNE 6, 2025

AWS Glue is a widely-used serverless data integration service that uses automated extract, transform, and load ( ETL ) methods to prepare data for analysis. It offers a simple and efficient solution for data processing in organizations. being data exactly matches the classifier, and 0.0

AWS 66

AWS Scala Metadata Data Lake 66

Engineering at Meta

JANUARY 22, 2025

In this blog, we will delve into an early stage in PAI implementation: data lineage. Data lineage refers to the process of tracing the journey of data as it moves through various systems, illustrating how data transitions from one data asset, such as a database table (the source asset), to another (the sink asset).

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

ProjectPro

JUNE 6, 2025

Snowflake provides data warehousing, processing, and analytical solutions that are significantly quicker, simpler to use, and more adaptable than traditional systems. Snowflake is not based on existing database systems or big data software platforms like Hadoop.

Architecture 56

Architecture IT Data Warehouse Amazon Web Services 56

ProjectPro

JUNE 6, 2025

Data engineering is the foundation for data science and analytics by integrating in-depth knowledge of data technology, reliable data governance and security, and a solid grasp of data processing. Data engineers need to meet various requirements to build data pipelines.

AWS 52

AWS Data Engineer Data Engineering Project 52

ProjectPro

JUNE 6, 2025

The role of a data engineer is to use tools for interacting with the database management systems. These data pipelines are fundamental to any organization that wants to source data organized and efficiently. for working on cloud data warehouses. You will work with unstructured data and NoSQL relational databases.

Data Engineer 58

Data Engineer Data Engineering Engineering Amazon Web Services 58

ProjectPro

JUNE 6, 2025

The field of data engineering is focused on ensuring that data is accessible, reliable, and easily processed by other teams within an organization, such as data analysts and data scientists. It involves various technical skills, including database design, data modeling, and ETL (Extract, Transform, Load) processes.

Data Engineer 49

Data Engineer Data Engineering Engineering Hadoop 49

ProjectPro

JUNE 6, 2025

With the global data volume projected to surge from 120 zettabytes in 2023 to 181 zettabytes by 2025, PySpark's popularity is soaring as it is an essential tool for efficient large scale data processing and analyzing vast datasets. They are distributed across the cluster, enabling efficient data processing at scale.

Hadoop 68

Hadoop Metadata Java Datasets 68

ProjectPro

JUNE 6, 2025

You can contribute to Apache Beam open-source big data project here: [link] 2. Clickhouse Source: Github Clickhouse is a column-oriented database management system used for the online analytical processing of queries ( also known as OLAP). DataFrames are used by Spark SQL to accommodate structured and semi-structured data.

Big Data 49

Big Data Project Metadata Programming Language 49

ProjectPro

JUNE 6, 2025

And, out of these professions, we will focus on the data engineering job role in this blog and list out a comprehensive list of projects to help you prepare for the same. Cloud computing skills, especially in Microsoft Azure, SQL , Python , and expertise in big data technologies like Apache Spark and Hadoop, are highly sought after.

Data Engineer 59

Data Engineer Data Engineering Project Engineering 59

Cloudera

DECEMBER 3, 2024

Rich set of SQL (query, DDL, DML) commands: Create or manipulate database objects, run queries, load and modify data, perform time travel operations, and convert Hive external tables to Iceberg tables using SQL commands. Create Database and Tables: Open HUE and execute the following to create a database and tables.

Metadata 84

Metadata SQL Data Warehouse Database 84

ProjectPro

JUNE 6, 2025

Data engineering has become crucial to any modern organization's technology stack. The need for fast and efficient data processing is high, as companies increasingly rely on data to make business decisions and improve product quality. But what books should you read if you want to learn more about data engineering?

Data Engineer 45

Data Engineer Data Engineering Engineering Lambda Architecture 45

ProjectPro

JUNE 6, 2025

These tools are crucial in modern business intelligence and data-driven decision-making processes. They provide a centralized repository for data, known as a data warehouse, where information from disparate sources like databases, spreadsheets, and external systems can be integrated.

Data Storage 58

Data Storage PostgreSQL Data Warehouse AWS 58

ProjectPro

JUNE 6, 2025

But, before we delve into the specifics of these tools, let's establish a foundational understanding of what a data pipeline is and why it is indispensable in data analytics and business intelligence. What are Data Pipelines? How Do Data Pipelines Work? Pros Messages are replicated across multiple brokers for durability.

Data Pipeline 49

Data Pipeline Google Cloud AWS Kafka 49

ProjectPro

JUNE 6, 2025

The demand for data-related roles has increased massively in the past few years. Companies are actively seeking talent in these areas, and there is a huge market for individuals who can manipulate data, work with large databases and build machine learning algorithms. Have you thought about what happens when more data comes in?

Engineering 75

Engineering Software Engineer Software Engineering Deep Learning 75

ProjectPro

JUNE 6, 2025

The two most popular AWS data engineering services for processing data at scale for analytics operations are Amazon EMR and AWS Glue. EMR is a more powerful big data processing solution to provide real-time data streaming for machine learning applications. Let's compare AWS Glue vs. You typically pay $0.44

Big Data 52

Big Data AWS Amazon Web Services Project 52

Cloudera

DECEMBER 4, 2024

All this by making it easier for customers to connect their workloads with Snowflake, Cloudera, and unique AWS services such as Amazon Simple Storage Service (Amazon S3), Amazon Elastic Kubernetes Service (Amazon EKS) , Amazon Relational Database Service (Amazon RDS), Amazon Elastic Compute Cloud (Amazon EC2), Amazon EMR and Amazon Athena.

AWS 89

AWS Raw Data Relational Database Government 89

ProjectPro

JUNE 6, 2025

Graduating from ETL Developer to Data Engineer Career transitions come with challenges. Suppose you are already working in the data industry as an ETL developer. You can easily transition to other data-driven jobs such as data engineer , analyst, database developer, and scientist.

Data Engineer 59

Data Engineer Data Engineering Engineering ETL Tools 59

ProjectPro

JUNE 6, 2025

Physical data model- The physical data model includes all necessary tables, columns, relationship constraints, and database attributes for physical database implementation. A physical model's key parameters include database performance, indexing approach, and physical storage. It makes data more accessible.

Data Warehouse 53

Data Warehouse NoSQL PostgreSQL Relational Database 53

ProjectPro

JUNE 6, 2025

Apache Hive and Apache Spark are the two popular Big Data tools available for complex data processing. To effectively utilize the Big Data tools, it is essential to understand the features and capabilities of the tools. Hive is built on top of Hadoop and provides the measures to read, write, and manage the data.

Hadoop 45

Hadoop Java Big Data Tools SQL 45

Snowflake

FEBRUARY 13, 2025

Out-of-the-box business continuity/disaster recovery: Snowflake enables customers to easily safeguard mission-critical accounts and data sets to maintain uptime. It's easy to use, there's no maintenance, and database administration is drastically reduced. It gives us functionality we can't get anywhere else and it costs us less.

Management 93

Management Government Cloud Unstructured Data 93

Snowflake

APRIL 9, 2025

KAWA combines analytics, automation and AI agents to help enterprises build data apps and AI workflows quickly and achieve their digital transformation goals. It connects structured and unstructured databases across sources and uses a no-code UI or Python for advanced and predictive analytics.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Ascend.io

NOVEMBER 14, 2024

Why Future-Proofing Your Data Pipelines Matters Data has become the backbone of decision-making in businesses across the globe. The ability to harness and analyze data effectively can make or break a company’s competitive edge. Set Up Auto-Scaling: Configure auto-scaling for your data processing and storage resources.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Data Integration 52

ProjectPro

JUNE 6, 2025

This section will cover the most commonly asked questions for an Amazon Data Engineer interview. Candidates should focus on Data Modelling , ETL Processes, Data Warehousing, Big Data Technologies, Programming Skills, AWS services, data processing technologies, and real-world problem-solving scenarios.

Data Engineer 40

Data Engineer Data Engineering Engineering NoSQL 40

ProjectPro

JUNE 6, 2025

A data architect, in turn, understands the business requirements, examines the current data structures, and develops a design for building an integrated framework of easily accessible, safe data aligned with business strategy. Table of Contents What is a Data Architect Role?

Data Architect 40

Data Architect Data Mining Programming Language Java 40

ProjectPro

JUNE 6, 2025

If you are still wondering whether or why you need to master SQL for data engineering, read this blog to take a deep dive into the world of SQL for data engineering and how it can take your data engineering skills to the next level. Data engineers can perform any quality checks using the DDL commands in SQL.

Data Engineer 45

Data Engineer Data Engineering SQL Engineering 45

ProjectPro

JUNE 6, 2025

So, when we lift and shift SSIS packages to the data factory, we use Azure SSIS Integration Runtime. What is required to execute an SSIS package in Data Factory? We must create an SSIS integration runtime and an SSISDB catalog hosted in the Azure SQL server database or Azure SQL-managed instance before executing an SSIS package.

Data Lake 60

Data Lake Metadata SQL Datasets 60

ProjectPro

JUNE 6, 2025

If you're looking to revolutionize your data processing and analysis, Python for ETL is the key to unlock the door. Check out this ultimate guide to explore the fascinating world of ETL with Python and discover why it's the top choice for modern data enthusiasts. Python ETL really empowers you to transform data like a pro.

Python 40

Python ETL Tools Data Warehouse Programming Language 40

ProjectPro

JUNE 6, 2025

For example, a cloud architect might enroll in a data engineering course to learn how to design and implement data pipelines using cloud services. Gaining such expertise can streamline data processing, ensuring data is readily available for analytics and decision-making.

Data Engineer 45

Data Engineer Data Engineering Cloud Engineering 45

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineer 248

Data Engineer Data Engineering Engineering Data 248

ProjectPro

JUNE 6, 2025

With the Talend big data tool , Talend developers can quickly create an environment for on-premise or cloud data integration tasks that work well with Spark, Apache Hadoop , and NoSQL databases. Companies that use enterprise solutions yet experience longer computation times often choose the Talend Big Data approach.

ETL Tools 45

ETL Tools Big Data Java Metadata 45

ProjectPro

JUNE 6, 2025

So, have you been wondering what happens to all the data collected from different sources, logs on your machine, data generated from your mobile, data in databases, customer data, and so on? We can do a lot of data analysis and produce visualizations to deliver value from these data sources.

Data Science 40

Data Science Data Engineer Data Engineering Engineering 40

ProjectPro

JUNE 6, 2025

Extraction- Data is extracted from multiple sources such as databases, applications, or files. Transformation- After extraction, the data undergoes transformation- cleaned, standardized, and modified to match the desired format. data warehouses). and, finally, loading (storing) it in a central location (e.g.,

Data Integration 40

Data Integration Project Data Lake Hospitality 40

ProjectPro

JUNE 6, 2025

ETL developer is a software developer who uses various tools and technologies to design and implement data integration processes across an organization. The role of an ETL developer is to extract data from multiple sources, transform it into a usable format and load it into a data warehouse or any other destination database.

ETL Tools 40

ETL Tools Data Cleanse Data Warehouse Big Data 40