Data Engineering Weekly

JULY 28, 2024

[link] Mikkel Dengsøe: How top data teams are structured What would the correct ratio of the number of data engineers corresponding to the overall people count in an organization? What are the different roles inside the data engineering functions, and what should their ratio be?

Data Engineer 109

Data Engineer Data Engineering Engineering Database-centric 109

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.

Data Engineer 130

Data Engineer Data Engineering Engineering Programming Language 130

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.

Data Engineer 130

Data Engineer Data Engineering Engineering Programming Language 130

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Process 217

Data Process Data Engineer Data Engineering Process 217

Jesse Anderson

DECEMBER 12, 2022

Big data projects were given to data scientists and data warehouse teams, where the projects subsequently failed. As clearly evident as that sounds now, my writing about needing data engineering went heavily against the grain of everything that was written at the time. Now people are excited about Rust.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineer 52

Data Engineer Data Engineering Python Engineering 52

Confessions of a Data Guy

FEBRUARY 26, 2023

Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. At least it’s an excuse to […] The post AWS Lambdas – Python vs Rust.

AWS 356

AWS Python Data Engineer Data Engineering 356

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. Your user base has quintupled, analytics requests are piling up, and that trusty Python script crashes more often than it runs. Back then, it worked just fine for your small team and handful of customers.

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Confessions of a Data Guy

APRIL 16, 2023

You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.

Data Engineer 147

Data Engineer Data Engineering Engineering SQL 147

Towards Data Science

NOVEMBER 20, 2024

It is worth mentioning that this article comes from an Electrical and Electronic Engineer graduate who went all the way and spent almost 8 years in academia learning about the Energy sector (and when I say all the way, I mean from a bachelor degree to a PhD and postdoc). Similarly, data engineering positions have seen a 98% increase.

Data Engineer 61

Data Engineer Data Engineering Engineering Electronics 61

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineer Data Engineering Engineering 147

Waitingforcode

FEBRUARY 3, 2023

In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. The part for Python will follow next week! We all have our habits and as programmers, libraries and frameworks are definitely a part of the group.

Scala 130

Scala Java Data Engineer Data Engineering 130

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Confessions of a Data Guy

SEPTEMBER 9, 2023

Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.

Data Engineer 162

Data Engineer Data Engineering Engineering Data Pipeline 162

Data Engineering Weekly

OCTOBER 13, 2024

Genie addresses these challenges by providing quick and accurate answers to questions, retrieving relevant information from internal knowledge bases, and reducing the need for constant interaction with on-call engineers.

Data Engineer 52

Data Engineer Data Engineering Engineering PostgreSQL 52

Data Engineering Weekly

AUGUST 4, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. We look at efficiency in data processing within a single node, which increases the momentum for systems like DuckDB, Arrow, and Polaris.

Data Engineer 80

Data Engineer Data Engineering Engineering Data 80

Confessions of a Data Guy

OCTOBER 6, 2023

I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling […] The post The Ultimate Data Engineering Chadstack. appeared first on Confessions of a Data Guy. At the moment Rust and Airflow are at least somewhere at the top of that list. Running Rust inside Apache Airflow.

Data Engineer 147

Data Engineer Data Engineering Engineering Data 147

Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here.

Data Engineer 125

Data Engineer Data Engineering Engineering Pipeline-centric 125

Towards Data Science

MAY 22, 2023

Solving data preparation tasks with ChatGPT Photo by Ricardo Gomez Angel on Unsplash Data engineering makes up a large part of the data science process. In CRISP-DM this process stage is called “data preparation”. It comprises tasks such as data ingestion, data transformation and data quality assurance.

Data Engineer 89

Data Engineer Data Engineering Engineering Data Science 89

Data Engineering Weekly

MAY 26, 2024

link] Akshay Agrawal: Lessons learned reinventing the Python notebook I wrote about the need for a Notebook-like orchestration system: A Notebook is all I want or Don't. A closer look at the OBTs Star Schema vs OBT (One Big Table) data modeling is another ongoing debate in data engineering practitioners.

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Seattle Data Guy

NOVEMBER 19, 2024

Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Of course, there are many challenges… Read more The post Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python appeared first on Seattle Data Guy.

Python 130

Python Data Data Science Data Engineering 130

Data Engineering Weekly

MARCH 17, 2024

It also introduces emerging standards like the Open Data Contract Standard and Data Product Descriptor Specification. As you know, I’m fascinated by data products and the potential to change the data engineering practice. Can we measure the cost of data incidents?

Data Engineer 70

Data Engineer Data Engineering Engineering Kafka 70

Data Engineering Weekly

FEBRUARY 18, 2024

Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. link] Sponsored: Data modeling and exploration in Playground 2.0 Elevate your data skills!

Data Engineer 62

Data Engineer Data Engineering Engineering Data 62

Data Engineering Weekly

JULY 14, 2024

Streaming execution to process a small chunk of data at a time. link] Ibis: Querying 1TB on a laptop with Python dataframes Ibis is an open-source dataframe framework to interact with multiple database engines. Intermediate spilling to disk while computing aggregations. The buffer manager caches as many pages as possible.

Data Engineer 98

Data Engineer Data Engineering Engineering Unstructured Data 98

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Data Engineering Podcast

JANUARY 30, 2022

Summary Pandas is a powerful tool for cleaning, transforming, manipulating, or enriching data, among many other potential uses. As a result it has become a standard tool for data engineers for a wide range of applications. What are the main tasks that you have seen Pandas used for in a data engineering context?

Data Engineer 100

Data Engineer Data Engineering Engineering Python 100

Towards Data Science

JULY 26, 2024

Learn from Software Engineers and Discover the Joy of ‘Worse is Better’ Thinking source: unsplash.com Recently, I have had the fortune of speaking to a number of data engineers and data architects about the problems they face with data in their businesses. That was it. So, What the Heck is Radical Simplicity?

Data Engineer 57

Data Engineer Data Engineering Engineering Data Warehouse 57

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineer 78

Data Engineer Data Engineering Engineering SQL 78

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Team Data Science

JANUARY 8, 2021

The purpose of this post is to expose you to the skills needed as a data engineer; now let’s look into them Understand the fundamental skill Recently, functions of computer engineering have become more important in organizations that are handling vast volumes of data, including data in diverse formats.

Data Engineer 130

Data Engineer Data Engineering Engineering Unstructured Data 130

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineer 98

Data Engineer Data Engineering Coding Project 98

Towards Data Science

JUNE 12, 2024

Python tricks and techniques for data ingestion, validation, processing, and testing: a practical walkthrough Continue reading on Towards Data Science »

Python 51

Python Data Engineer Data Engineering Coding 51

Snowflake

JUNE 20, 2024

In the age of AI, enterprises are increasingly looking to extract value from their data at scale but often find it difficult to establish a scalable data engineering foundation that can process the large amounts of data required to build or improve models. Snowflake customers see an average of 4.6x

Data Engineer 94

Data Engineer Data Engineering Scala Engineering 94

Cloudyard

SEPTEMBER 2, 2024

Read Time: 1 Minute, 36 Second Snowflake’s support for Python stored procedures allows data engineers and scientists to leverage Python’s vast ecosystem directly within Snowflake. This capability enables advanced analytics, custom data processing, and seamless integration of Python libraries.

Python 96

Python Utilities Coding Data Engineer 96

Towards Data Science

OCTOBER 21, 2023

Advanced ETL techniques for beginners Continue reading on Towards Data Science »

Data Engineer 82

Data Engineer Data Engineering Python Engineering 82