Analytics Vidhya

AUGUST 4, 2024

Introduction Managing complicated, interrelated information is more important than ever in today’s data-driven society. Traditional databases, while still valuable, often falter when it comes to handling highly connected data. Enter the unsung heroes of the data world: graph databases.

Database 213

Database Data Engineering Data Engineer Engineering 213

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Process 217

Data Process Data Engineering Data Engineer Process 217

Confessions of a Data Guy

FEBRUARY 7, 2023

Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.

Data Engineering 221

Data Engineering Data Engineer Engineering Data 221

Start Data Engineering

APRIL 22, 2024

Most data projects use Docker to set up the data infra locally (and often in production). Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1. Introduction Docker can be overwhelming to start with.

Data Engineering 225

Data Engineering Data Engineer Engineering Data 225

Analytics Vidhya

JUNE 24, 2023

He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.

Data Engineering 204

Data Engineering Data Engineer Engineering Data 204

Analytics Vidhya

JULY 25, 2023

In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.

Data Engineering 200

Data Engineering Data Engineer Engineering Data 200

Seattle Data Guy

JANUARY 17, 2023

What is the state of data infra? Are data engineers all learning Rust? Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Is everyone switching to DuckDB?

Data Engineering 211

Data Engineering Data Engineer Engineering Data 211

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Introduction 2. Objective 3. Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7.

Data Engineering 246

Data Engineering Data Engineer Project Data Lake 246

KDnuggets

DECEMBER 4, 2024

Learn to build, run, and manage data engineering pipelines both locally and in the cloud using popular tools.

Data Engineering 152

Data Engineering Data Engineer Engineering Project 152

KDnuggets

APRIL 12, 2024

The only data engineering roadmap you need for an introduction to concepts, tools, and techniques to collect, store, transform, analyze, and model data.

Data Engineering 152

Data Engineering Data Engineer Engineering Data 152

Start Data Engineering

MAY 30, 2024

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python 147

Python Data Engineering Data Engineer Engineering 147

Confessions of a Data Guy

APRIL 28, 2024

The funny thing is, at the time, and today, it […] The post Why did Golang lose to Rust for Data Engineering? appeared first on Confessions of a Data Guy. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about.

Data Engineering 162

Data Engineering Data Engineer Engineering Data 162

KDnuggets

FEBRUARY 12, 2024

Interested in data engineering but don't know where to start? Get up to speed in data engineering fundamentals with this free course.

Data Engineering 141

Data Engineering Data Engineer Engineering Data 141

KDnuggets

AUGUST 30, 2024

Data engineering is best learned by doing projects. Here are six projects focusing on different data engineering skills to ensure you have it all covered. But which ones?

Data Engineering 139

Data Engineering Data Engineer Project Engineering 139

KDnuggets

JULY 15, 2024

Is it possible to learn data engineering for free? I claim it is and present the evidence for that in the form of 10 free data engineering courses.

Data Engineering 154

Data Engineering Data Engineer Engineering Certification 154

KDnuggets

APRIL 25, 2024

Interested in switching to data engineering? Here’s a list of Python libraries you’ll find super helpful.

Python 160

Python Data Engineering Data Engineer Engineering 160

Start Data Engineering

AUGUST 13, 2024

Data structures and algorithms to know 2.1. Introduction 2. Dictionary 2.3. Counter (from collections module) 2.7. Graph search 2.8.1 Depth First Search (DFS) 2.8.2. Breadth First Search BFS 2.9. Binary Search 3. Common DSA questions asked during DE interviews 3.1. Intervals 3.

Algorithm 200

Algorithm Data Engineering Data Engineer Engineering 200

KDnuggets

OCTOBER 3, 2024

Learn the data engineering tools for data orchestration, database management, batch processing, ETL (Extract, Transform, Load), data transformation, data visualization, and data streaming.

Data Engineering 145

Data Engineering Data Engineer Engineering Data 145

KDnuggets

MAY 21, 2024

Learn data engineering through free courses, tutorials, books, tools, guides, roadmaps, practice exercises, projects, and other resources.

Data Engineering 144

Data Engineering Data Engineer Engineering Data 144

KDnuggets

NOVEMBER 30, 2023

Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company.

Data Engineering 149

Data Engineering Data Engineer Engineering Data Workflow 149

KDnuggets

SEPTEMBER 2, 2024

Interested in data engineering? Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks.

Python 152

Python Data Engineering Data Engineer Engineering 152

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.

Data Engineering 162

Data Engineering Data Engineer Engineering Data Pipeline 162

KDnuggets

JANUARY 26, 2024

Data Engineering ZoomCamp offers free access to reading materials, video tutorials, assignments, homeworks, projects, and workshops.

Data Engineering 156

Data Engineering Data Engineer Engineering Data 156

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 119

Data Engineering Data Engineer Engineering Generalist 119

Seattle Data Guy

APRIL 19, 2024

How data is structured, managed and processed will continue to grow in importance as the demand for AI and machine learning increase. It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Pipeline 130

Start Data Engineering

SEPTEMBER 4, 2024

Key parts of data systems: 2.1. Data flow design 2.3. Data processing design 2.5. Data storage design 2.7. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools. Introduction 2. Requirements 2.2.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Storage 130

Confessions of a Data Guy

APRIL 30, 2024

The post Reaction to Data Engineering Survey for 2024 appeared first on Confessions of a Data Guy.

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

Confessions of a Data Guy

AUGUST 7, 2024

One thing I find myself doing these days (I am unsure how I feel about this), is teaching others to solve problems … Data Engineering problems to be specific. I […] The post How to Solve Data Engineering Problems appeared first on Confessions of a Data Guy.

Data Engineering 113

Data Engineering Data Engineer Engineering Data 113

Seattle Data Guy

AUGUST 8, 2024

If you work in data, then AI is everywhere at this point. But whether AI is hype or reality doesn’t change the fact that data engineers will play a major role in ensuring that the data sets that are utilized for the growing use cases are usable both by machines and humans.

Data Engineering 130

Data Engineering Data Engineer Engineering Utilities 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Confessions of a Data Guy

FEBRUARY 26, 2023

While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. appeared first on Confessions of a Data Guy. Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. Performance and Cost Savings.

AWS 356

AWS Python Data Engineering Data Engineer 356

databricks

MAY 1, 2024

The generative AI revolution is transforming the way that teams work, and Databricks Assistant leverages the best of these advancements. It allows you.

Data Engineering 132

Data Engineering Data Engineer Engineering Data 132

databricks

JUNE 13, 2024

Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines.

Data Engineering 144

Data Engineering Data Engineer Engineering Data Pipeline 144

Simon Späti

JANUARY 3, 2023

I used VS Code, Sublime, Notepad++, TextMate, and others, but the shortcut with cmd(+shift)+end, jumping with option+arrow-keys from word to word, needed to be faster at some point. I was hitting my limits. Everything I was doing I did decently fast, but I didn’t get any faster. Vim is the only editor you get faster with time.

Data Engineering 195

Data Engineering Data Engineer Engineering Coding 195

KDnuggets

FEBRUARY 28, 2024

Begin your data professional journey from the basics of statistics to building a production-grade AI application.

Machine Learning 160

Machine Learning Data Science Data Engineering Data Engineer 160

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. The team at Zenlytic have leaned on the promise of large language models to build an AI agent that lets you converse with your data. Data lakes are notoriously complex.

Building 278

Building Data Lake High Quality Data Business Intelligence 278