Start Data Engineering

SEPTEMBER 4, 2024

Data storage design 2.7. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools. Monitoring & Alerting 2.9. Infrastructure 3. Conclusion 1.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Storage 130

Confessions of a Data Guy

SEPTEMBER 13, 2024

Did you know there are only 3 types of Data Engineers? The post The 3 Types of Data Engineers. appeared first on Confessions of a Data Guy. It’s true. I hope you are the right one.

Data Engineering 100

Data Engineering Data Engineer Engineering Data 100

Analytics Vidhya

APRIL 3, 2023

Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers. Nevertheless, setting up a streaming data pipeline to power such dashboards may […] The post Data Engineering for Streaming Data on GCP appeared first on Analytics Vidhya.

Data Engineering 251

Data Engineering Data Engineer Engineering Data 251

Confessions of a Data Guy

AUGUST 7, 2024

One thing I find myself doing these days (I am unsure how I feel about this), is teaching others to solve problems … Data Engineering problems to be specific. I […] The post How to Solve Data Engineering Problems appeared first on Confessions of a Data Guy.

Data Engineering 113

Data Engineering Data Engineer Engineering Data 113

Analytics Vidhya

JUNE 24, 2023

He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.

Data Engineering 204

Data Engineering Data Engineer Engineering Data 204

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineering 240

Data Engineering Data Engineer Engineering Data 240

Analytics Vidhya

JULY 25, 2023

In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.

Data Engineering 200

Data Engineering Data Engineer Engineering Data 200

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 117

Data Engineering Data Engineer Engineering Generalist 117

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Process 217

Data Process Data Engineering Data Engineer Process 217

Confessions of a Data Guy

APRIL 28, 2024

The funny thing is, at the time, and today, it […] The post Why did Golang lose to Rust for Data Engineering? appeared first on Confessions of a Data Guy. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about.

Data Engineering 162

Data Engineering Data Engineer Engineering Data 162

KDnuggets

OCTOBER 3, 2024

Learn the data engineering tools for data orchestration, database management, batch processing, ETL (Extract, Transform, Load), data transformation, data visualization, and data streaming.

Data Engineering 127

Data Engineering Data Engineer Engineering Data 127

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. The blog Prompt Engineering for a Better SQL Code Generation With LLMs is a pretty good guide on applying prompt engineering to improve productivity.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Seattle Data Guy

APRIL 19, 2024

It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Pipeline 130

Seattle Data Guy

AUGUST 8, 2024

But whether AI is hype or reality doesn’t change the fact that data engineers will play a major role in ensuring that the data sets that are utilized for the growing use cases are usable both by machines and humans.

Data Engineering 130

Data Engineering Data Engineer Engineering Utilities 130

KDnuggets

AUGUST 30, 2024

Data engineering is best learned by doing projects. Here are six projects focusing on different data engineering skills to ensure you have it all covered. But which ones?

Data Engineering 108

Data Engineering Data Engineer Project Engineering 108

Start Data Engineering

APRIL 22, 2024

Most data projects use Docker to set up the data infra locally (and often in production). Introduction 2. Docker concepts 2.1. Define the OS and its configurations with an image 2.2. Use the image to run containers 2.2.1. Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1.

Data Engineering 225

Data Engineering Data Engineer Engineering Data 225

Data Engineering Weekly

JUNE 24, 2024

Bessemer publishes the Data + AI infrastructure market map to help companies understand the landscape and the key players. link] Shreya Shankar: Short Musings on AI Engineering and "Failed AI Projects" LLMs significantly reduce the entry barrier and are faster than ever for prototypes. Are there enough usecases?

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

KDnuggets

JULY 15, 2024

Is it possible to learn data engineering for free? I claim it is and present the evidence for that in the form of 10 free data engineering courses.

Data Engineering 141

Data Engineering Data Engineer Engineering Certification 141

Jesse Anderson

SEPTEMBER 14, 2023

There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. How can we use LLMs in data engineering?

Data Engineering 147

Data Engineering Data Engineer Engineering NoSQL 147

Confessions of a Data Guy

FEBRUARY 7, 2023

Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.

Data Engineering 221

Data Engineering Data Engineer Engineering Data 221

Jesse Anderson

DECEMBER 12, 2022

Apache Spark came in 2009 and gave a unified batch and streaming engine. Apache Flink came in 2011 and gave us our first real streaming engine. Apache Kafka came in 2011 and gave the industry a much better way to move real-time data. DJ Patil coined the term Data Scientist in 2008. We lacked a scalable pub/sub system.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Confessions of a Data Guy

APRIL 16, 2023

You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.

Data Engineering 147

Data Engineering Data Engineer Engineering SQL 147

Data Engineering Weekly

OCTOBER 27, 2024

We recently published a comprehensive engineering guide to build a privacy-first design. impactdatasummit.com Uber: Streamlining Financial Precision - Uber’s Advanced Settlement Accounting System Possibly one of the complicated pipelines to build is the Financial reconciliation engine. link] Alibaba: Evolution of Flink 2.0

Data Engineering 57

Data Engineering Data Engineer Engineering Unstructured Data 57

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineering Data Engineer Engineering 147

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineering 103

Data Engineering Data Engineer Engineering Kafka 103

Data Engineering Weekly

JUNE 30, 2024

Engineering jobs saw a 10.42% decline. The authors emphasize the increasing need for high-quality data for training and inference, focusing on unstructured data pipelines, retrieval-augmented generation (RAG), data curation, and AI memory. Software, app, and web development jobs decreased by 20.62%.

Data Engineering 102

Data Engineering Data Engineer Engineering Unstructured Data 102

Seattle Data Guy

JANUARY 17, 2023

Are data engineers all learning Rust? Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Are Snowflake and Databricks still fighting over total cost of ownership?

Data Engineering 211

Data Engineering Data Engineer Engineering Data 211

Analytics Vidhya

SEPTEMBER 20, 2023

Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise.

Data Engineering 130

Data Engineering Data Engineer Coding Project 130

KDnuggets

SEPTEMBER 2, 2024

Interested in data engineering? Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks.

Python 141

Python Data Engineering Data Engineer Engineering 141

Jesse Anderson

JANUARY 29, 2024

The premier of my latest talk covering The State of Data Engineering. This starts with data warehousing and goes into data science. I finish off by showing how data engineering can avoid the same fate as data warehousing and data science.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Science 130

Data Engineering Weekly

NOVEMBER 3, 2024

The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. Sampling is an obvious strategy for data size, but the layered approach and dynamic inclusion of dependencies are some key techniques I learned with the case study.

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62

KDnuggets

APRIL 12, 2024

The only data engineering roadmap you need for an introduction to concepts, tools, and techniques to collect, store, transform, analyze, and model data.

Data Engineering 129

Data Engineering Data Engineer Engineering Data 129

Seattle Data Guy

MAY 20, 2023

Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools.

Data Engineering 147

Data Engineering Data Engineer Project Engineering 147

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Support Data Engineering Podcast

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225