ProjectPro

JUNE 6, 2025

You can easily connect to multiple data sources, manipulate data, and load it into different data storage systems using Python. This makes it an ideal choice for ETL developers, data engineers , and data analysts, even those without a strong programming background.

Python 40

Python ETL Tools Data Warehouse Programming Language 40

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. Hadoop Platform Hadoop is an open-source software library created by the Apache Software Foundation.

Data Engineer 98

Data Engineer Data Engineering Engineering Pipeline-centric 98

ProjectPro

JUNE 6, 2025

The ETL (Extract, Transform, Load) process follows four main steps: i) Connect and Collect: Connect to the data source/s and move data to local and crowdsource data storage. ii) Data transformation using computing services such as HDInsight, Hadoop , Spark, etc.

Data Lake 60

Data Lake Metadata SQL Datasets 60

AltexSoft

MARCH 14, 2023

Batch jobs are often scheduled to load data into the warehouse, while real-time data processing can be achieved using solutions like Apache Kafka and Snowpipe by Snowflake to stream data directly into the cloud warehouse. But this distinction has been blurred with the era of cloud data warehouses.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Monte Carlo

MAY 14, 2024

The “legacy” table formats The data landscape has evolved so quickly that table formats pioneered within the last 25 years are already achieving “legacy” status. It was designed to support high-volume data exchange and compatibility across different system versions, which is essential for streaming architectures such as Apache Kafka.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Knowledge Hut

NOVEMBER 2, 2023

An Azure Data Engineer is a professional who is in charge of designing, implementing, and maintaining data processing systems and solutions on the Microsoft Azure cloud platform. A Data Engineer is responsible for designing the entire architecture of the data flow while taking the needs of the business into account.

Data Engineer 52

Data Engineer Data Engineering Coding Project 52

Knowledge Hut

MARCH 28, 2024

Job Role 1: Azure Data Engineer Azure Data Engineers develop, deploy, and manage data solutions with Microsoft Azure data services. They use many data storage, computation, and analytics technologies to develop scalable and robust data pipelines.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Warehouse 52

AltexSoft

SEPTEMBER 21, 2023

The Elastic Stacks Elasticsearch is integral within analytics stacks, collaborating seamlessly with other tools developed by Elastic to manage the entire data workflow — from ingestion to visualization. Elastic Certified Analyst : Aimed at professionals using Kibana for data visualization.

Engineering 52

Engineering NoSQL Java Programming Language 52

Knowledge Hut

SEPTEMBER 29, 2023

Why Should You Get an Azure Data Engineer Certification? Becoming an Azure data engineer allows you to seamlessly blend the roles of a data analyst and a data scientist. One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role.

Certification 52

Certification Data Engineer Data Engineering Engineering 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? This is frequently referred to as a 5 or 7 layer (depending on who you ask) data stack like in the image below.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

ProjectPro

JUNE 6, 2025

Web Server Log Processing In this project, you'll process web server logs using a combination of Hadoop, Flume, Spark, and Hive on Azure. Starting with setting up an Azure Virtual Machine, you'll install necessary big data tools and configure Flume agents for log data ingestion.

Project 64

Project Transportation Data Pipeline Data Ingestion 64

Christophe Blefari

SEPTEMBER 28, 2023

The era of Big Data was characterised by Hadoop, HDFS, distributed computing (Spark), above the JVM. That's why big data technologies got swooshed by the modern data stack when it arrived on the market—excepting Spark. We need to store, process and visualise data, everything else is just marketing.

Cloud Storage 147

Cloud Storage Hadoop Big Data SQL 147