Start Data Engineering

APRIL 22, 2024

Most data projects use Docker to set up the data infra locally (and often in production). Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1. Introduction Docker can be overwhelming to start with.

Data Engineering 225

Data Engineering Data Engineer Engineering Data 225

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Process 217

Data Process Data Engineering Data Engineer Process 217

Analytics Vidhya

AUGUST 4, 2024

Introduction Managing complicated, interrelated information is more important than ever in today’s data-driven society. Traditional databases, while still valuable, often falter when it comes to handling highly connected data. Enter the unsung heroes of the data world: graph databases.

Database 213

Database Data Engineering Data Engineer Engineering 213

Confessions of a Data Guy

FEBRUARY 7, 2023

Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.

Data Engineering 221

Data Engineering Data Engineer Engineering Data 221

Analytics Vidhya

JULY 25, 2023

In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.

Data Engineering 200

Data Engineering Data Engineer Engineering Data 200

Analytics Vidhya

JUNE 24, 2023

He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.

Data Engineering 204

Data Engineering Data Engineer Engineering Data 204

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Introduction 2. Objective 3. Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7.

Data Engineering 246

Data Engineering Data Engineer Project Data Lake 246

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Seattle Data Guy

JANUARY 17, 2023

What is the state of data infra? Are data engineers all learning Rust? Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Is everyone switching to DuckDB?

Data Engineering 211

Data Engineering Data Engineer Engineering Data 211

Start Data Engineering

MAY 30, 2024

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python 147

Python Data Engineering Data Engineer Engineering 147

Start Data Engineering

AUGUST 13, 2024

Data structures and algorithms to know 2.1. Introduction 2. Dictionary 2.3. Counter (from collections module) 2.7. Graph search 2.8.1 Depth First Search (DFS) 2.8.2. Breadth First Search BFS 2.9. Binary Search 3. Common DSA questions asked during DE interviews 3.1. Intervals 3.

Algorithm 200

Algorithm Data Engineering Data Engineer Engineering 200

KDnuggets

DECEMBER 4, 2024

Learn to build, run, and manage data engineering pipelines both locally and in the cloud using popular tools.

Data Engineering 143

Data Engineering Data Engineer Engineering Project 143

Confessions of a Data Guy

APRIL 28, 2024

The funny thing is, at the time, and today, it […] The post Why did Golang lose to Rust for Data Engineering? appeared first on Confessions of a Data Guy. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about.

Data Engineering 162

Data Engineering Data Engineer Engineering Data 162

KDnuggets

JULY 15, 2024

Is it possible to learn data engineering for free? I claim it is and present the evidence for that in the form of 10 free data engineering courses.

Data Engineering 146

Data Engineering Data Engineer Engineering Certification 146

KDnuggets

APRIL 12, 2024

The only data engineering roadmap you need for an introduction to concepts, tools, and techniques to collect, store, transform, analyze, and model data.

Data Engineering 135

Data Engineering Data Engineer Engineering Data 135

KDnuggets

APRIL 25, 2024

Interested in switching to data engineering? Here’s a list of Python libraries you’ll find super helpful.

Python 152

Python Data Engineering Data Engineer Engineering 152

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.

Data Engineering 162

Data Engineering Data Engineer Engineering Data Pipeline 162

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 118

Data Engineering Data Engineer Engineering Generalist 118

KDnuggets

OCTOBER 3, 2024

Learn the data engineering tools for data orchestration, database management, batch processing, ETL (Extract, Transform, Load), data transformation, data visualization, and data streaming.

Data Engineering 135

Data Engineering Data Engineer Engineering Data 135

KDnuggets

JANUARY 26, 2024

Data Engineering ZoomCamp offers free access to reading materials, video tutorials, assignments, homeworks, projects, and workshops.

Data Engineering 155

Data Engineering Data Engineer Engineering Data 155

KDnuggets

MAY 21, 2024

Learn data engineering through free courses, tutorials, books, tools, guides, roadmaps, practice exercises, projects, and other resources.

Data Engineering 133

Data Engineering Data Engineer Engineering Data 133

Start Data Engineering

SEPTEMBER 4, 2024

Key parts of data systems: 2.1. Data flow design 2.3. Data processing design 2.5. Data storage design 2.7. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools. Introduction 2. Requirements 2.2.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Storage 130

KDnuggets

SEPTEMBER 2, 2024

Interested in data engineering? Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks.

Python 146

Python Data Engineering Data Engineer Engineering 146

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Confessions of a Data Guy

APRIL 30, 2024

The post Reaction to Data Engineering Survey for 2024 appeared first on Confessions of a Data Guy.

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

KDnuggets

AUGUST 30, 2024

Data engineering is best learned by doing projects. Here are six projects focusing on different data engineering skills to ensure you have it all covered. But which ones?

Data Engineering 118

Data Engineering Data Engineer Project Engineering 118

Confessions of a Data Guy

AUGUST 7, 2024

One thing I find myself doing these days (I am unsure how I feel about this), is teaching others to solve problems … Data Engineering problems to be specific. I […] The post How to Solve Data Engineering Problems appeared first on Confessions of a Data Guy.

Data Engineering 113

Data Engineering Data Engineer Engineering Data 113

Analytics Vidhya

DECEMBER 30, 2024

What will data engineering look like in 2025? How will generative AI shape the tools and processes Data Engineers rely on today? As the field evolves, Data Engineers are stepping into a future where innovation and efficiency take center stage.

100

100

Seattle Data Guy

AUGUST 8, 2024

If you work in data, then AI is everywhere at this point. But whether AI is hype or reality doesn’t change the fact that data engineers will play a major role in ensuring that the data sets that are utilized for the growing use cases are usable both by machines and humans.

Data Engineering 130

Data Engineering Data Engineer Engineering Utilities 130

Start Data Engineering

SEPTEMBER 18, 2024

Parts of data engineering 3.1. Identify what tool to use to process data 3.3. Data flow architecture 3. Introduction 2. Requirements 3.1.1. Understand input datasets available 3.1.2. Define what the output dataset will look like 3.1.3. Define SLAs so stakeholders know what to expect 3.1.4.

Project 240

Project Building Datasets Architecture 240

Confessions of a Data Guy

FEBRUARY 26, 2023

While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. appeared first on Confessions of a Data Guy. Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. Performance and Cost Savings.

AWS 356

AWS Python Data Engineering Data Engineer 356

databricks

JUNE 13, 2024

Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines.

Data Engineering 141

Data Engineering Data Engineer Engineering Data Pipeline 141

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Data lakes are notoriously complex.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Data Engineering Podcast

SEPTEMBER 10, 2023

Summary Data systems are inherently complex and often require integration of multiple technologies. This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. With Materialize, you can!

BI 208

BI SQL Machine Learning Data 208

KDnuggets

FEBRUARY 28, 2024

Begin your data professional journey from the basics of statistics to building a production-grade AI application.

Machine Learning 157

Machine Learning Data Science Data Engineering Data Engineer 157

Start Data Engineering

OCTOBER 17, 2024

Handy functions for common data processing scenarios 1.1. STRUCT data types are sorted based on their keys from left to right 1.4. Introduction Setup SQL tips 1. Need to filter on WINDOW function without CTE/Subquery use QUALIFY 1.2. Need the first/last row in a partition, use DISTINCT ON 1.3.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

Simon Späti

JANUARY 3, 2023

I used VS Code, Sublime, Notepad++, TextMate, and others, but the shortcut with cmd(+shift)+end, jumping with option+arrow-keys from word to word, needed to be faster at some point. I was hitting my limits. Everything I was doing I did decently fast, but I didn’t get any faster. Vim is the only editor you get faster with time.

Data Engineering 195

Data Engineering Data Engineer Engineering Coding 195