Maxime Beauchemin

JANUARY 20, 2017

Unlike data scientists — and inspired by our more mature parent, software engineering  — data engineers build tools, infrastructure, frameworks, and services. In fact, it’s arguable that data engineering is much closer to software engineering than it is to a data science.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is the role of a Data Engineer? Data scientists and data Analysts depend on data engineers to build these data pipelines.

Data Engineering 98

Data Engineering Data Engineer Engineering Hadoop 98

Christophe Blefari

APRIL 8, 2023

I was in the Hadoop world and all I was doing was denormalisation. The only normalisation I did was back at the engineering school while learning SQL with Normal Forms. I still firmly believe that this is not the role of a data engineer. Data modeling should not be a required data engineer skill.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

I was in the Hadoop world and all I was doing was denormalisation. The only normalisation I did was back at the engineering school while learning SQL with Normal Forms. I still firmly believe that this is not the role of a data engineer. Data modeling should not be a required data engineer skill.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Confluent

SEPTEMBER 25, 2019

Here, the bank loan business division has essentially become software. Of course, this is not to imply that companies will become only software (there are still plenty of people in even the most software-centric companies), just that the full scope of the business is captured in an integrated software defined process.

Database-centric 56

Database-centric Kafka Pipeline-centric Retail 56

Knowledge Hut

FEBRUARY 27, 2023

Data engineering is all about building, designing, and optimizing systems for acquiring, storing, accessing, and analyzing data at scale. Data engineering builds data pipelines for core professionals like data scientists, consumers, and data-centric applications. Who is Data Engineer, and What Do They Do?

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. 7 Be Intentional About the Batching Model in Your Data Pipelines Different batching models. Test system with A/A test.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52