Start Data Engineering

OCTOBER 22, 2021

SQL skills 2.1. Data modeling 2.1.1. Data storage 2.2. Data transformation 2.2.1. Data pipeline 2.4. Data analytics 3. Introduction SQL is the bread and butter of data engineering. Introduction 2. Gathering requirements 2.1.2. Exploration 2.1.3. Modeling 2.1.4. Query planner 2.2.3.

SQL 130

SQL Data Engineer Data Engineering Engineering 130

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineer 103

Data Engineer Data Engineering Engineering Kafka 103

Jesse Anderson

DECEMBER 12, 2022

They were the first companies to commercialize open source big data technologies and pushed the marketing and commercialization of Hadoop. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. With an immutable file system like HDFS, we needed scalable databases to read and write data randomly.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

KDnuggets

FEBRUARY 24, 2023

This article will discuss SQL visualization, its role in augmenting the modern-day data engineer, and five categories of SQL visualization tools.

SQL 102

SQL Data Engineer Data Engineering Engineering 102

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineer 105

Data Engineer Data Engineering Engineering Metadata 105

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is the role of the data engineer in supporting those interfaces?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Towards Data Science

NOVEMBER 20, 2024

It is worth mentioning that this article comes from an Electrical and Electronic Engineer graduate who went all the way and spent almost 8 years in academia learning about the Energy sector (and when I say all the way, I mean from a bachelor degree to a PhD and postdoc). Similarly, data engineering positions have seen a 98% increase.

Data Engineer 61

Data Engineer Data Engineering Engineering Electronics 61

Data Engineering Weekly

OCTOBER 6, 2024

link] Ravi Vedula: Leveraging AI for next-level data productivity in IDEAS In the new era of LLM, data engineers strive to improve our interaction with the vast data storage in the data warehouse. “text-to-SQL” and “text-to-insight.” Sonnet, GPT-4o, Mistral Large 2, Llama 3.1

Data Engineer 62

Data Engineer Data Engineering Engineering Data Warehouse 62

KDnuggets

FEBRUARY 27, 2023

5 SQL Visualization Tools for Data Engineers • Free TensorFlow 2.0 Complete Course • The Importance of Probability in Data Science • 4 Ways to Rename Pandas Columns • 5 Statistical Paradoxes Data Scientists Should Know

SQL 129

SQL Data Engineer Data Engineering Engineering 129

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. While the Python API connector remains available for specific SQL use cases, the new API is designed to be your go-to solution.

Data Pipeline 111

Data Pipeline Python Data Engineer Data Engineering 111

Data Engineering Weekly

SEPTEMBER 22, 2024

Uber: QueryGPT – Natural Language to SQL Using Generative AI Companies are increasingly adopting the natural language interface to interact with the enterprise data. link] Sponsored: IMPACT - Have you seen the agenda for IMPACT: The Data Observability Summit yet?

Data Engineer 70

Data Engineer Data Engineering Engineering Data Lake 70

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. You’re left wondering if there’s a breaking point where your DIY data solution won’t cut it anymore—and honestly, you might be there already. It means you’re scaling!

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Data Engineering Weekly

SEPTEMBER 15, 2024

During his keynote, he’ll share his insights into how data and pattern recognition shape the world, build trust, and—sometimes—even help us predict the future. The blog narrates how the lack of neutral guidance or standards leads to ambiguity and the formation of a central team to bring trust in data.

Data Engineer 67

Data Engineer Data Engineering Engineering Data Lake 67

Data Engineering Weekly

APRIL 7, 2024

VCs have moved on from data catalogs, yet practitioners told me they look forward to solving data discovery. link] Pinterest: How we built Text-to-SQL at Pinterest Last week Intuit shared its key learning building Text 2 SQL , and Pinterest publishes the tech deep dive on how its internal Text2SQL work.

Data Engineer 70

Data Engineer Data Engineering Engineering Unstructured Data 70

Data Engineering Weekly

OCTOBER 13, 2024

Genie addresses these challenges by providing quick and accurate answers to questions, retrieving relevant information from internal knowledge bases, and reducing the need for constant interaction with on-call engineers.

Data Engineer 52

Data Engineer Data Engineering Engineering PostgreSQL 52

Data Engineering Weekly

MAY 26, 2024

[link] Ryan Eakman: SQLFrame - Turning PySpark into a Universal DataFrame API SQL or DataFrames, two programming models often used interchangeability in engineering data. It is a long standing question on people wondering In what situations should you use SQL instead of Pandas as a data scientist?

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Data Engineering Weekly

JULY 7, 2024

The authors implement this using Airflow's dynamic task mapping to automatically clone tagged models to each contributor's development dataset, allowing seamless dbt builds with production-like data. The blog explains how we can programmatically measure the real cost of Azure databricks SQL warehouse instances.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Weekly

SEPTEMBER 29, 2024

link] Google: SQL Has Problems - We Can Fix Them - Pipe Syntax In SQL It was a good weekend read about the proposed pipe syntax in SQL, which is more similar to Unix pipes in terms of its core concept—sequential data flow and transformation. Unix pipes typically represent a physical flow of data between processes.

Data Engineer 62

Data Engineer Data Engineering Engineering SQL 62

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineer 78

Data Engineer Data Engineering Engineering SQL 78

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. While this idea isn’t completely foreign to data engineering, it can certainly be described as an extremely uncommon practice. Data is different. It’s terrifying. I’m afraid so.

Data Engineer 95

Data Engineer Data Engineering Engineering Software Engineer 95

Data Engineering Weekly

JULY 14, 2024

[link] Wasteman.codes: Engineering Principles for Building Financial Systems Any system that involves moving money or counting money is always complex to get it correct. I used to joke the entire Walstreet runs on unknown SQL codes that no one understands.

Data Engineer 98

Data Engineer Data Engineering Engineering Unstructured Data 98

Start Data Engineering

OCTOBER 11, 2021

Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Introduction Skills 1. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11.

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Team Data Science

DECEMBER 17, 2020

Even then, developing a team to successfully manage AI tasks is essential to tackle any challenges faced by organizations as regard data. While studies over the last few years have shown that data engineering, a vital phase forward to Data Science and AI, it is still a phase that many businesses completely skip, misunderstand, or neglect.

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Confessions of a Data Guy

FEBRUARY 21, 2024

New SQL Practice Problems I’m trying something new. I get a lot of questions from folks about getting into the Data Engineering space, how to get better, grow, learn, etc. SQL Practice Problems. Some moons ago I wrote a Data Engineering Practice repo on GitHub for free, and some 1.2K

SQL 100

SQL Data Engineer Data Engineering Engineering 100

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Data Engineering Weekly

DECEMBER 18, 2023

Lorin Hochstein: “Human error” means they don’t understand how the system worked The post is not directly related to Data Engineering but system operations in general. I included this post because I often see high-pitched LinkedIn posts stating it is the human fault, especially around data quality issues.

Data Engineer 67

Data Engineer Data Engineering Engineering Food 67

Knowledge Hut

MAY 3, 2023

With companies increasingly relying on data-driven insights to make informed decisions, there has never been a greater need for skilled specialists who can manage and evaluate vast amounts of data. The roles of data analyst and data engineer have emerged as two of the most in-demand professions in today's job market.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Cleanse 98

Data Engineering Weekly

DECEMBER 3, 2023

link] Microsoft: Generative AI for Beginners Understanding Gen-AI becomes a mandatory skill for application developers and data engineers. One of the highlights for me in the blog is the next step of ClickHouse adoption; as a believer in bringing “query to the data” rather than “data to query,” I love it.

Data Engineer 70

Data Engineer Data Engineering Engineering Bytes 70

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineer 52

Data Engineer Data Engineering Python Engineering 52

Team Data Science

JANUARY 8, 2021

The purpose of this post is to expose you to the skills needed as a data engineer; now let’s look into them Understand the fundamental skill Recently, functions of computer engineering have become more important in organizations that are handling vast volumes of data, including data in diverse formats.

Data Engineer 130

Data Engineer Data Engineering Engineering Unstructured Data 130

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. I wish more data engineers would start sharing their on-call experience soon.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Pipeline 52

Data Engineering Podcast

JULY 10, 2022

Summary Building and maintaining reliable data assets is the prime directive for data engineers. While it is easy to say, it is endlessly complex to implement, requiring data professionals to be experts in a wide range of disparate topics while designing and implementing complex topologies of information workflows.

Data Engineer 100

Data Engineer Data Engineering Engineering MongoDB 100

Data Engineering Weekly

FEBRUARY 4, 2024

Joe Reis: Definition of Data Modeling & What Data Modeling Is not Joe raised a very fundamental question in data engineering. What is Data Modeling, and what is not? He rightly points out that many data engineers will name any of the modeling techniques as data modeling.

Data Engineer 62

Data Engineer Data Engineering Engineering PostgreSQL 62

RandomTrees

FEBRUARY 2, 2024

GPT-based data engineering accelerators make the working of data more accessible. These accelerators use GPT models to do data tasks faster, fix any issues, and save a lot of time. GPT models change data in simple language and also provide summaries and explanations. One can rely on this information.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Pipeline 52

Data Engineering Weekly

SEPTEMBER 8, 2024

link] Tangbao: Flink SQL Development Experience Sharing It is very rare to see open and honest learning about using technology, and this time, the author shares the development of Flink SQL. link] Tom Moertel: Sampling with SQL Sampling is one of the most powerful tools for extracting meaning from large datasets.

Data Engineer 52

Data Engineer Data Engineering Engineering Kafka 52