Start Data Engineering

MAY 30, 2024

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python 147

Python Data Engineering Data Engineer Engineering 147

KDnuggets

AUGUST 30, 2024

Data engineering is best learned by doing projects. Here are six projects focusing on different data engineering skills to ensure you have it all covered. But which ones?

Data Engineer 120

Data Engineer Data Engineering Project Engineering 120

Confessions of a Data Guy

AUGUST 7, 2024

One thing I find myself doing these days (I am unsure how I feel about this), is teaching others to solve problems … Data Engineering problems to be specific. I […] The post How to Solve Data Engineering Problems appeared first on Confessions of a Data Guy.

Data Engineer 114

Data Engineer Data Engineering Engineering Data 114

Seattle Data Guy

JANUARY 17, 2023

With all the recent data events I have put together I inevitably run into new data engineers who are either finishing up college or looking to transition into a data engineer or data scientist position. In fact I have talked to several newly graduated engineers who are struggling to find work.

Data Engineering 223

Data Engineering Data Engineer Engineering Data 223

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineer 251

Data Engineer Data Engineering Engineering Data 251

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineer 114

Data Engineer Data Engineering Engineering Generalist 114

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Analytics Vidhya

JULY 25, 2023

In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.

Data Engineering 200

Data Engineering Data Engineer Engineering Data 200

Analytics Vidhya

JUNE 24, 2023

He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.

Data Engineering 204

Data Engineering Data Engineer Engineering Data 204

Start Data Engineering

APRIL 22, 2024

Most data projects use Docker to set up the data infra locally (and often in production). Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1. Introduction Docker can be overwhelming to start with.

Data Engineer 225

Data Engineer Data Engineering Engineering Data 225

Confessions of a Data Guy

APRIL 28, 2024

The funny thing is, at the time, and today, it […] The post Why did Golang lose to Rust for Data Engineering? appeared first on Confessions of a Data Guy. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about.

Data Engineer 163

Data Engineer Data Engineering Engineering Data 163

Seattle Data Guy

AUGUST 8, 2024

If you work in data, then AI is everywhere at this point. But whether AI is hype or reality doesn’t change the fact that data engineers will play a major role in ensuring that the data sets that are utilized for the growing use cases are usable both by machines and humans.

Data Engineering 130

Data Engineering Data Engineer Engineering Utilities 130

KDnuggets

SEPTEMBER 2, 2024

Interested in data engineering? Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks.

Python 149

Python Data Engineering Data Engineer Engineering 149

Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications.

Data Engineer 240

Data Engineer Data Engineering Engineering Data 240

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Process 217

Data Process Data Engineering Data Engineer Process 217

Data Engineering Weekly

JULY 28, 2024

link] Chip Huyan: Building A Generative AI Platform We can’t deny that Gen-AI is becoming an integral part of product strategy, pushing the need for platform engineering. Switching from Apache Spark to Ray improves compact 12X larger datasets than Apache Spark, improves cost efficiency by 91%, and processes 13X more data per hour.

Data Engineer 103

Data Engineer Data Engineering Engineering Database-centric 103

Data Engineering Weekly

SEPTEMBER 22, 2024

Editor’s Note: Coming Next on Comparison Matrix Series - Data LakeHouse Our mission is to empower data professionals and organizations to make informed, data-driven decisions by providing a comprehensive buyer's guide and comparison matrix for selecting the best data tools.

Data Engineer 70

Data Engineer Data Engineering Engineering Data Lake 70

KDnuggets

JULY 15, 2024

Is it possible to learn data engineering for free? I claim it is and present the evidence for that in the form of 10 free data engineering courses.

Data Engineering 149

Data Engineering Data Engineer Engineering Certification 149

Seattle Data Guy

APRIL 19, 2024

How data is structured, managed and processed will continue to grow in importance as the demand for AI and machine learning increase. It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Pipeline 130

Start Data Engineering

AUGUST 13, 2024

Data structures and algorithms to know 2.1. Introduction 2. Dictionary 2.3. Counter (from collections module) 2.7. Graph search 2.8.1 Depth First Search (DFS) 2.8.2. Breadth First Search BFS 2.9. Binary Search 3. Common DSA questions asked during DE interviews 3.1. Intervals 3.

Algorithm 200

Algorithm Data Engineer Data Engineering Engineering 200

Data Engineering Weekly

SEPTEMBER 15, 2024

Uber: DataMesh - How Uber laid the foundations for the Data Lake Cloud migration Many companies are slowly adopting DataMesh, and Uber writes about adopting the data mesh principle. Whether or not Data Mesh is a separate product is debatable, but it is certainly an impactful framework for scaling data platforms.

Data Engineer 67

Data Engineer Data Engineering Engineering Data Lake 67

KDnuggets

APRIL 12, 2024

The only data engineering roadmap you need for an introduction to concepts, tools, and techniques to collect, store, transform, analyze, and model data.

Data Engineer 141

Data Engineer Data Engineering Engineering Data 141

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70

Confessions of a Data Guy

FEBRUARY 7, 2023

Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.

Data Engineer 221

Data Engineer Data Engineering Engineering Data 221

Jesse Anderson

SEPTEMBER 14, 2023

There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. How can we use LLMs in data engineering?

Data Engineer 147

Data Engineer Data Engineering Engineering NoSQL 147

Jesse Anderson

JANUARY 29, 2024

The premier of my latest talk covering The State of Data Engineering. This starts with data warehousing and goes into data science. I finish off by showing how data engineering can avoid the same fate as data warehousing and data science.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Science 130

Data Engineering Weekly

SEPTEMBER 2, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineer 67

Data Engineer Data Engineering Engineering Entertainment 67

Confessions of a Data Guy

AUGUST 1, 2024

The post CI/CD for Data Engineers. appeared first on Confessions of a Data Guy.

Data Engineering 100

Data Engineering Data Engineer Engineering Data 100

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly. Apache Spark came in 2009 and gave a unified batch and streaming engine.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Confessions of a Data Guy

APRIL 16, 2023

You might think […] The post DuckDB vs Polars for Data Engineering. appeared first on Confessions of a Data Guy. I haven’t seen this since Databricks and Snowflake first came out and started throwing mud at each other.

Data Engineer 148

Data Engineer Data Engineering Engineering SQL 148

KDnuggets

FEBRUARY 12, 2024

Interested in data engineering but don't know where to start? Get up to speed in data engineering fundamentals with this free course.

Data Engineer 132

Data Engineer Data Engineering Engineering Data 132

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineering Data Engineer Engineering 147

Seattle Data Guy

JANUARY 17, 2023

What is the state of data infra? Are data engineers all learning Rust? Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Is everyone switching to DuckDB?

Data Engineering 211

Data Engineering Data Engineer Engineering Data 211

KDnuggets

MAY 21, 2024

Learn data engineering through free courses, tutorials, books, tools, guides, roadmaps, practice exercises, projects, and other resources.

Data Engineer 134

Data Engineer Data Engineering Engineering Data 134

Confessions of a Data Guy

APRIL 30, 2024

The post Reaction to Data Engineering Survey for 2024 appeared first on Confessions of a Data Guy.

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

Analytics Vidhya

SEPTEMBER 20, 2023

Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise.

Data Engineering 130

Data Engineering Data Engineer Coding Project 130

Seattle Data Guy

MAY 20, 2023

Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools.

Data Engineering 147

Data Engineering Data Engineer Project Engineering 147