Analytics Vidhya

AUGUST 4, 2024

These powerful tools are designed to manage and query intricate data relationships effortlessly. This article discusses […] The post Neo4j vs. Amazon Neptune: Graph Databases in Data Engineering appeared first on Analytics Vidhya.

Database 213

Database Data Engineering Data Engineer Engineering 213

Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Process 217

Data Process Data Engineering Data Engineer Process 217

Analytics Vidhya

JUNE 24, 2023

He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.

Data Engineering 204

Data Engineering Data Engineer Engineering Data 204

Confessions of a Data Guy

FEBRUARY 7, 2023

Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.

Data Engineering 221

Data Engineering Data Engineer Engineering Data 221

Analytics Vidhya

JULY 25, 2023

In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.

Data Engineering 200

Data Engineering Data Engineer Engineering Data 200

Seattle Data Guy

JANUARY 17, 2023

Are data engineers all learning Rust? Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Are Snowflake and Databricks still fighting over total cost of ownership?

Data Engineering 211

Data Engineering Data Engineer Engineering Data 211

Start Data Engineering

APRIL 22, 2024

Most data projects use Docker to set up the data infra locally (and often in production). Introduction 2. Docker concepts 2.1. Define the OS and its configurations with an image 2.2. Use the image to run containers 2.2.1. Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1.

Data Engineering 225

Data Engineering Data Engineer Engineering Data 225

KDnuggets

DECEMBER 4, 2024

Learn to build, run, and manage data engineering pipelines both locally and in the cloud using popular tools.

Data Engineering 152

Data Engineering Data Engineer Engineering Project 152

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Support Data Engineering Podcast

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

KDnuggets

APRIL 12, 2024

The only data engineering roadmap you need for an introduction to concepts, tools, and techniques to collect, store, transform, analyze, and model data.

Data Engineering 152

Data Engineering Data Engineer Engineering Data 152

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Introduction 2. Objective 3. Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7.

Data Engineering 246

Data Engineering Data Engineer Project Data Lake 246

KDnuggets

FEBRUARY 12, 2024

Interested in data engineering but don't know where to start? Get up to speed in data engineering fundamentals with this free course.

Data Engineering 141

Data Engineering Data Engineer Engineering Data 141

Confessions of a Data Guy

APRIL 28, 2024

The funny thing is, at the time, and today, it […] The post Why did Golang lose to Rust for Data Engineering? appeared first on Confessions of a Data Guy. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about.

Data Engineering 162

Data Engineering Data Engineer Engineering Data 162

KDnuggets

AUGUST 30, 2024

Data engineering is best learned by doing projects. Here are six projects focusing on different data engineering skills to ensure you have it all covered. But which ones?

Data Engineering 139

Data Engineering Data Engineer Project Engineering 139

Start Data Engineering

MAY 30, 2024

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python 147

Python Data Engineering Data Engineer Engineering 147

KDnuggets

JULY 15, 2024

Is it possible to learn data engineering for free? I claim it is and present the evidence for that in the form of 10 free data engineering courses.

Data Engineering 154

Data Engineering Data Engineer Engineering Certification 154

KDnuggets

APRIL 25, 2024

Interested in switching to data engineering? Here’s a list of Python libraries you’ll find super helpful.

Python 160

Python Data Engineering Data Engineer Engineering 160

KDnuggets

MAY 21, 2024

Learn data engineering through free courses, tutorials, books, tools, guides, roadmaps, practice exercises, projects, and other resources.

Data Engineering 144

Data Engineering Data Engineer Engineering Data 144

KDnuggets

OCTOBER 3, 2024

Learn the data engineering tools for data orchestration, database management, batch processing, ETL (Extract, Transform, Load), data transformation, data visualization, and data streaming.

Data Engineering 145

Data Engineering Data Engineer Engineering Data 145

KDnuggets

SEPTEMBER 2, 2024

Interested in data engineering? Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks.

Python 152

Python Data Engineering Data Engineer Engineering 152

KDnuggets

NOVEMBER 30, 2023

Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company.

Data Engineering 149

Data Engineering Data Engineer Engineering Data Workflow 149

KDnuggets

JANUARY 26, 2024

Data Engineering ZoomCamp offers free access to reading materials, video tutorials, assignments, homeworks, projects, and workshops.

Data Engineering 156

Data Engineering Data Engineer Engineering Data 156

Confessions of a Data Guy

SEPTEMBER 9, 2023

Nothing screams “why are flying by night,” than coming into a Data Team only to find no tests, no docs, no deployments, no Docker, no nothing. […] The post The Role of DevOps and CI/CD in Data Engineering appeared first on Confessions of a Data Guy.

Data Engineering 162

Data Engineering Data Engineer Engineering Data Pipeline 162

Data Engineering Weekly

JULY 21, 2024

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

Data Engineering 119

Data Engineering Data Engineer Engineering Generalist 119

Seattle Data Guy

APRIL 19, 2024

It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Pipeline 130

Start Data Engineering

AUGUST 13, 2024

Data structures and algorithms to know 2.1. Introduction 2. Dictionary 2.3. Counter (from collections module) 2.7. Graph search 2.8.1 Depth First Search (DFS) 2.8.2. Breadth First Search BFS 2.9. Binary Search 3. Common DSA questions asked during DE interviews 3.1. Intervals 3.

Algorithm 200

Algorithm Data Engineering Data Engineer Engineering 200

Confessions of a Data Guy

APRIL 30, 2024

The post Reaction to Data Engineering Survey for 2024 appeared first on Confessions of a Data Guy.

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

Start Data Engineering

SEPTEMBER 4, 2024

Data storage design 2.7. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools. Monitoring & Alerting 2.9. Infrastructure 3. Conclusion 1.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Storage 130

Confessions of a Data Guy

AUGUST 7, 2024

One thing I find myself doing these days (I am unsure how I feel about this), is teaching others to solve problems … Data Engineering problems to be specific. I […] The post How to Solve Data Engineering Problems appeared first on Confessions of a Data Guy.

Data Engineering 113

Data Engineering Data Engineer Engineering Data 113

Seattle Data Guy

AUGUST 8, 2024

But whether AI is hype or reality doesn’t change the fact that data engineers will play a major role in ensuring that the data sets that are utilized for the growing use cases are usable both by machines and humans.

Data Engineering 130

Data Engineering Data Engineer Engineering Utilities 130

databricks

MAY 1, 2024

The generative AI revolution is transforming the way that teams work, and Databricks Assistant leverages the best of these advancements. It allows you.

Data Engineering 132

Data Engineering Data Engineer Engineering Data 132

Confessions of a Data Guy

FEBRUARY 26, 2023

While it raised the ire of many a Python Data Engineer, I thought it sounded like a great idea. appeared first on Confessions of a Data Guy. Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. Performance and Cost Savings.

AWS 356

AWS Python Data Engineering Data Engineer 356

databricks

JUNE 13, 2024

Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines.

Data Engineering 144

Data Engineering Data Engineer Engineering Data Pipeline 144

Simon Späti

JANUARY 3, 2023

I used VS Code, Sublime, Notepad++, TextMate, and others, but the shortcut with cmd(+shift)+end, jumping with option+arrow-keys from word to word, needed to be faster at some point. I was hitting my limits. Everything I was doing I did decently fast, but I didn’t get any faster. Vim is the only editor you get faster with time.

Data Engineering 195

Data Engineering Data Engineer Engineering Coding 195

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. It’s the big blueprint we data engineers follow in order to transform raw data into valuable insights.

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

KDnuggets

FEBRUARY 28, 2024

Begin your data professional journey from the basics of statistics to building a production-grade AI application.

Machine Learning 160

Machine Learning Data Science Data Engineering Data Engineer 160

Start Data Engineering

OCTOBER 17, 2024

Handy functions for common data processing scenarios 1.1. STRUCT data types are sorted based on their keys from left to right 1.4. Introduction Setup SQL tips 1. Need to filter on WINDOW function without CTE/Subquery use QUALIFY 1.2. Need the first/last row in a partition, use DISTINCT ON 1.3.

SQL 130

SQL Data Engineering Data Engineer Engineering 130