Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

Confessions of a Data Guy

FEBRUARY 26, 2023

Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.

AWS 356

AWS Python Data Engineering Data Engineer 356

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Towards Data Science

NOVEMBER 20, 2024

It is worth mentioning that this article comes from an Electrical and Electronic Engineer graduate who went all the way and spent almost 8 years in academia learning about the Energy sector (and when I say all the way, I mean from a bachelor degree to a PhD and postdoc). Similarly, data engineering positions have seen a 98% increase.

Data Engineering 62

Data Engineering Data Engineer Engineering Electronics 62

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Process 217

Data Process Data Engineering Data Engineer Process 217

Jesse Anderson

DECEMBER 12, 2022

Apache Spark came in 2009 and gave a unified batch and streaming engine. Apache Flink came in 2011 and gave us our first real streaming engine. Apache Kafka came in 2011 and gave the industry a much better way to move real-time data. DJ Patil coined the term Data Scientist in 2008. We lacked a scalable pub/sub system.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

JULY 2, 2023

Summary Feature engineering is a crucial aspect of the machine learning workflow. In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Engineering 130

Waitingforcode

FEBRUARY 3, 2023

In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. The part for Python will follow next week! We all have our habits and as programmers, libraries and frameworks are definitely a part of the group.

Scala 130

Scala Java Data Engineering Data Engineer 130

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. Your user base has quintupled, analytics requests are piling up, and that trusty Python script crashes more often than it runs. Back then, it worked just fine for your small team and handful of customers.

Data Engineering 52

Data Engineering Data Engineer Building Engineering 52

Data Engineering Weekly

JULY 14, 2024

[link] Discord: How Discord Uses Open-Source Tools for Scalable Data Orchestration & Transformation Discord writes about its migration journey from a homegrown orchestration engine to Dagster. Real-world applications of Nexla’s RAG data flow capabilities in enhancing AI deployment.

Data Engineering 98

Data Engineering Data Engineer Engineering Unstructured Data 98

Confessions of a Data Guy

APRIL 16, 2023

You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

Data Engineering Weekly

AUGUST 4, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. The Archmage thrift proxy layer to hide the infrastructure and enable easier version upgrades is a good engineering practice that I would recommend for any platform team.

Data Engineering 80

Data Engineering Data Engineer Engineering Data 80

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineering Data Engineer Engineering 147

Data Engineering Weekly

OCTOBER 13, 2024

Genie addresses these challenges by providing quick and accurate answers to questions, retrieving relevant information from internal knowledge bases, and reducing the need for constant interaction with on-call engineers.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

Seattle Data Guy

NOVEMBER 19, 2024

Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Of course, there are many challenges… Read more The post Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python appeared first on Seattle Data Guy.

Python 130

Python Data Data Science Data Engineering 130

Towards Data Science

MAY 22, 2023

Solving data preparation tasks with ChatGPT Photo by Ricardo Gomez Angel on Unsplash Data engineering makes up a large part of the data science process. In CRISP-DM this process stage is called “data preparation”. It comprises tasks such as data ingestion, data transformation and data quality assurance.

Data Engineering 90

Data Engineering Data Engineer Engineering Data Science 90

Confessions of a Data Guy

SEPTEMBER 9, 2023

Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.

Data Engineering 162

Data Engineering Data Engineer Engineering Data Pipeline 162

Data Engineering Weekly

MAY 26, 2024

[link] Meta: Composable data management at Meta Meta writes about its transition to a composable data management system to improve interoperability, reusability, and engineering efficiency. It is a long standing question on people wondering In what situations should you use SQL instead of Pandas as a data scientist?

Data Engineering 76

Data Engineering Data Engineer Engineering Google Cloud 76

Confessions of a Data Guy

OCTOBER 6, 2023

I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack. appeared first on Confessions of a Data Guy. At the moment Rust and Airflow are at least somewhere at the top of that list. Running Rust inside Apache Airflow.

Data Engineering 147

Data Engineering Data Engineer Engineering Data 147

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineering 130

Data Engineering Data Engineer Engineering Cloud Computing 130

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here.

Data Engineering 125

Data Engineering Data Engineer Engineering Pipeline-centric 125

Data Engineering Weekly

MARCH 17, 2024

It also introduces emerging standards like the Open Data Contract Standard and Data Product Descriptor Specification. As you know, I’m fascinated by data products and the potential to change the data engineering practice. Can we measure the cost of data incidents?

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Cloudyard

SEPTEMBER 2, 2024

Read Time: 1 Minute, 36 Second Snowflake’s support for Python stored procedures allows data engineers and scientists to leverage Python’s vast ecosystem directly within Snowflake. This capability enables advanced analytics, custom data processing, and seamless integration of Python libraries.

Python 96

Python Utilities Coding Data Engineering 96

Data Engineering Podcast

JANUARY 30, 2022

Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. What are the main tasks that you have seen Pandas used for in a data engineering context?

Data Engineering 100

Data Engineering Data Engineer Engineering Python 100

Data Engineering Weekly

FEBRUARY 18, 2024

Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. link] Sponsored: Data modeling and exploration in Playground 2.0 Elevate your data skills!

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineering 79

Data Engineering Data Engineer Engineering SQL 79

Towards Data Science

JULY 26, 2024

Learn from Software Engineers and Discover the Joy of ‘Worse is Better’ Thinking source: unsplash.com Recently, I have had the fortune of speaking to a number of data engineers and data architects about the problems they face with data in their businesses. That was it. So, What the Heck is Radical Simplicity?

Data Engineering 58

Data Engineering Data Engineer Engineering Data Warehouse 58

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Towards Data Science

JUNE 12, 2024

Python tricks and techniques for data ingestion, validation, processing, and testing: a practical walkthrough Continue reading on Towards Data Science »

Python 52

Python Data Engineering Data Engineer Coding 52

Snowflake

JUNE 20, 2024

In the age of AI, enterprises are increasingly looking to extract value from their data at scale but often find it difficult to establish a scalable data engineering foundation that can process the large amounts of data required to build or improve models. Snowflake customers see an average of 4.6x

Data Engineering 94

Data Engineering Data Engineer Scala Engineering 94

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineering 98

Data Engineering Data Engineer Coding Project 98