ProjectPro

JUNE 6, 2025

You can use several datasets in this project covering various healthcare sources such as patient records, medical imaging data, electronic health records (EHRs), and hospital operational data. You will use Python libraries for data processing and transformation. This project enables you to do just that!

Data Integration 40

Data Integration Project Data Lake PostgreSQL 40

ProjectPro

JUNE 6, 2025

“Unlock the potential of your data with Azure Databricks: a unified analytics platform that combines the power of Apache Spark with the ease of Azure.” ” Azure Databricks is a fully managed service provided by Microsoft that offers the capabilities to create an open data lake house within the Azure cloud environment.

Data Engineering 40

Data Engineering Data Engineer Engineering Data Lake 40

ProjectPro

JUNE 6, 2025

AWS Data Pipeline AWS Data Pipeline is a cloud-based service by Amazon Web Services (AWS) that simplifies the orchestration of data workflows. It offers pre-built connectors for various AWS services, allowing users to seamlessly automate data movement and processing tasks within the AWS ecosystem.

Data Pipeline 40

Data Pipeline Building Retail Data Ingestion 40

ProjectPro

JUNE 6, 2025

Their role includes designing data pipelines, integrating data from multiple sources, and setting up databases and data lakes that can support machine learning and analytics workloads. They work with various tools and frameworks, such as Apache Spark, Hadoop , and cloud services, to manage massive amounts of data.

Data Engineering 40

Data Engineering Data Engineer Engineering ETL Tools 40

Data Engineering Podcast

JANUARY 1, 2022

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! To what extent do speed benchmarks inform decisions for modern data teams?

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

ProjectPro

JUNE 6, 2025

This can include tasks like data validation, data type conversion, deduplication, and aggregating data from different sources. Data Loading: The transformed data is loaded into a data warehouse or data lake, depending on the architecture of your data ecosystem.

Python 40

Python ETL Tools Data Warehouse Programming Language 40

Snowflake

JULY 22, 2024

LTIMindtree’s PolarSled Accelerator helps migrate existing legacy systems, such as SAP, Teradata and Hadoop, to Snowflake. top modernizing your data lake with Snowflake, watch our on demand webinar.

Government 119

Government Cloud Data Data Lake 119

Monte Carlo

MAY 14, 2024

The “legacy” table formats The data landscape has evolved so quickly that table formats pioneered within the last 25 years are already achieving “legacy” status. It was designed to support high-volume data exchange and compatibility across different system versions, which is essential for streaming architectures such as Apache Kafka.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

AltexSoft

MARCH 14, 2023

Built around a cloud data warehouse, data lake, or data lakehouse. Modern data stack tools are designed to integrate seamlessly with cloud data warehouses such as Redshift, Bigquery, and Snowflake, as well as data lakes or even the child of the first two — a data lakehouse.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Knowledge Hut

NOVEMBER 2, 2023

Azure Data Ingestion Pipeline Create an Azure Data Factory data ingestion pipeline to extract data from a source (e.g., Azure SQL Database, Azure Data Lake Storage). Data Aggregation Working with a sample of big data allows you to investigate real-time data processing, big data project design, and data flow.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Knowledge Hut

MARCH 28, 2024

Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data. Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

JUNE 14, 2023

5 Data pipeline architecture designs and their evolution The Hadoop era , roughly 2011 to 2017, arguably ushered in big data processing capabilities to mainstream organizations. Data then, and even today for some organizations, was primarily hosted in on-premises databases with non-scalable storage.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Knowledge Hut

MAY 23, 2024

Users can also leverage it for generating interactive visualizations over data. It also comes with lots of automation techniques that qualify users to eliminate manual data workflows. It can analyze data in real-time and can perform cluster management. It is much faster than other analytic workload tools like Hadoop.

Data Science 52

Data Science MongoDB Programming Language BI 52

Knowledge Hut

SEPTEMBER 29, 2023

Why Should You Get an Azure Data Engineer Certification? Becoming an Azure data engineer allows you to seamlessly blend the roles of a data analyst and a data scientist. One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

phData: Data Engineering

JANUARY 3, 2022

phData Cloud Foundation is dedicated to machine learning and data analytics, with prebuilt stacks for a range of analytical tools, including AWS EMR, Airflow, AWS Redshift, AWS DMS, Snowflake, Databricks, Cloudera Hadoop, and more. This required applying transformations and filters to the data for various business units.

IT 52

IT AWS Software Engineering Software Engineer 52

ProjectPro

JUNE 6, 2025

Web Server Log Processing In this project, you'll process web server logs using a combination of Hadoop, Flume, Spark, and Hive on Azure. Starting with setting up an Azure Virtual Machine, you'll install necessary big data tools and configure Flume agents for log data ingestion.

Project 64

Project Transportation Data Pipeline Datasets 64

AltexSoft

NOVEMBER 7, 2022

DevOps tasks — for example, creating scheduled backups and restoring data from them. Airflow is especially useful for orchestrating Big Data workflows. Airflow is not a data processing tool by itself but rather an instrument to manage multiple components of data processing. When Airflow won’t work.

PostgreSQL 52

PostgreSQL Metadata MySQL Python 52