Data Engineering Weekly

MAY 25, 2025

Dagster Components is now here Components provides a modular architecture that enables data practitioners to self-serve while maintaining engineering quality. Understanding this fact will help data tools break new ground with the advancement of AI agents. and Lite 2.0) to pinpoint drop-offs and high retention sections.

Data Engineer 62

Data Engineer Data Engineering Engineering PostgreSQL 62

Data Engineering Weekly

JUNE 1, 2025

Join Dagster and Neurospace to learn: - How to build AI pipelines with orchestration baked in - How to track data lineage for audits and traceability - Tips for designing compliant workflows under the EU AI Act Register for the technical session DuckDB: DuckLake - SQL as a Lakehouse Format DuckDB announced a new open table format, DuckLake.

Data Engineer 70

Data Engineer Data Engineering Engineering Relational Database 70

ProjectPro

JUNE 6, 2025

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. use SQL, compared to 61.7%

Data Engineer 45

Data Engineer Data Engineering SQL Engineering 45

Data Engineering Weekly

MAY 4, 2025

Get started → Editor’s Note: OpenXData Conference - 2025 - A Free Virtual Event A free virtual event on open data architectures - Iceberg, Hudi, lakehouses, query engines, and more. Talks from Netflix, dbt Labs, Databricks, Microsoft, Google, Meta, Peloton, and other open data geeks. May 21st, 9 am—3 pm PDT.

Data Engineer 67

Data Engineer Data Engineering Engineering Java 67

ProjectPro

JUNE 6, 2025

If you are planning to make a career transition into data engineering and want to know how to become a data engineer, this is the perfect place to begin your journey. Beginners will especially find it helpful if they want to know how to become a data engineer from scratch. in the following few sections.

Data Engineering 58

Data Engineering Data Engineer Engineering Amazon Web Services 58

ProjectPro

JUNE 6, 2025

Data engineering is the foundation for data science and analytics by integrating in-depth knowledge of data technology, reliable data governance and security, and a solid grasp of data processing. Data engineers need to meet various requirements to build data pipelines.

AWS 52

AWS Data Engineering Data Engineer Engineering 52

ProjectPro

JUNE 6, 2025

In the thought process of making a career transition from ETL developer to data engineer job roles? Read this blog to know how various data-specific roles, such as data engineer, data scientist, etc., Data analysis and visualization have traditionally been a common goal for businesses.

Data Engineer 59

Data Engineer Data Engineering Engineering ETL Tools 59

ProjectPro

JUNE 6, 2025

Most of us have observed that data scientist is usually labeled the hottest job of the 21st century, but is it the only most desirable job? No, that is not the only job in the data world. These trends underscore the growing demand and significance of data engineering in driving innovation across industries.

Data Engineering 59

Data Engineering Data Engineer Project Engineering 59

ProjectPro

JUNE 6, 2025

Over the past few years, data-related jobs have drastically increased. Previously, the spotlight was on gaining relevant insights from data, but recently, data handling has gained attention. Because of that, data engineer jobs have garnered recognition and popularity.

Data Engineering 49

Data Engineering Data Engineer Engineering Hadoop 49

ProjectPro

JUNE 6, 2025

In recent years, you must have seen a significant rise in businesses deploying data engineering projects on cloud platforms. These businesses need data engineers who can use technologies for handling data quickly and effectively since they have to manage potentially profitable real-time data.

Data Engineer 40

Data Engineer Data Engineering Engineering Google Cloud 40

ProjectPro

JUNE 6, 2025

This blog post provides an overview of the top 10 data engineering tools for building a robust data architecture to support smooth business operations. Table of Contents What are Data Engineering Tools? Dice Tech Jobs report 2020 indicates Data Engineering is one of the highest in-demand jobs worldwide.

Data Engineering 45

Data Engineering Data Engineer Engineering Kafka 45

ProjectPro

JUNE 6, 2025

The total amount of data that was created in 2020 was 64 zettabytes! The volume and the variety of data captured have also rapidly increased, with critical system sources such as smartphones, power grids, stock exchanges, and healthcare adding more data sources as the storage capacity increases.

Data Ingestion 45

Data Ingestion Data Engineer Data Engineering Project 45

databricks

JUNE 11, 2025

Events Data + AI Summit Data + AI World Tour Data Intelligence Days Event Calendar Blog and Podcasts Databricks Blog Explore news, product announcements, and more Databricks Mosaic Research Blog Discover the latest in our Gen AI research Data Brew Podcast Let’s talk data! REGISTER Ready to get started?

Entertainment 132

Entertainment Manufacturing Consulting Retail 132

ProjectPro

JUNE 6, 2025

Data engineering has become crucial to any modern organization's technology stack. The need for fast and efficient data processing is high, as companies increasingly rely on data to make business decisions and improve product quality. But what books should you read if you want to learn more about data engineering?

Data Engineering 45

Data Engineering Data Engineer Engineering Lambda Architecture 45

Data Engineering Weekly

DECEMBER 28, 2024

In this post, we delve into predictions for 2025, focusing on the transformative role of AI agents, workforce dynamics, and data platforms. For professionals across domains—data engineers, AI engineers, and data scientists—the message is clear: adapt or become obsolete.

Unstructured Data 112

Unstructured Data Metadata Government Data 112

Data Engineering Weekly

APRIL 27, 2025

Before Hoptimator, Pinot ingestion often required data producers to create and manage separate, Pinot-specific preprocessing jobs to optimize data, such as re-keying, filtering, and pre-aggregating. reducing user friction, operator toil, and resource consumption on Pinot servers, while automating pipeline management.

Data Engineer 62

Data Engineer Data Engineering Engineering Kafka 62

ProjectPro

JUNE 6, 2025

Becoming a data engineer can be challenging, but we are here to make the journey easier. In this blog, we have curated a list of the best data engineering courses so you can master this challenging field with confidence. Say goodbye to confusion and hello to a clear path to data engineering expertise!

Data Engineering 45

Data Engineering Data Engineer Cloud Engineering 45

ProjectPro

JUNE 6, 2025

This blog will help you understand what data engineering is with an exciting data engineering example, why data engineering is becoming the sexier job of the 21st century is, what is data engineering role, and what data engineering skills you need to excel in the industry, Table of Contents What is Data Engineering?

Data Science 40

Data Science Data Engineering Data Engineer Engineering 40

ProjectPro

JUNE 6, 2025

With over 175 full features service offerings, organizations are head hunting for AWS data engineers who can help them build and maintain the entire AWS cloud infrastructure to keep the applications up and running. Cloud platforms are becoming the new standard for managing an organization's data.

AWS 45

AWS Data Engineering Data Engineer Amazon Web Services 45

ProjectPro

JUNE 6, 2025

Modern businesses are migrating their on-premises data to cloud computing infrastructures to facilitate the digital business revolution. To achieve digital transformation, it is necessary to process, manage, and automate the vast volume of data that goes into the cloud platform.

Data Engineer 40

Data Engineer Data Engineering Project Engineering 40

ProjectPro

JUNE 6, 2025

Scala has been one of the most trusted and reliable programming languages for several tech giants and startups to develop and deploy their big data applications. Table of Contents What is Scala for Data Engineering? Why Should Data Engineers Learn Scala for Data Engineering?

Scala 40

Scala Data Engineering Data Engineer Engineering 40

ProjectPro

JUNE 6, 2025

Becoming a successful aws data engineer demands you to learn AWS for data engineering and leverage its various services for building efficient business applications. million organizations that want to be data-driven choose AWS as their cloud services partner. Table of Contents Why Learn AWS for Data Engineering?

AWS 40

AWS Data Engineer Data Engineering Engineering 40

ProjectPro

JUNE 6, 2025

Data engineering is gradually becoming the backbone of companies looking forward to leveraging data to improve business processes. This blog will discover how Python has become an integral part of implementing data engineering methods by exploring how to use Python for data engineering.

Data Engineering 40

Data Engineering Data Engineer Python Engineering 40

Christophe Blefari

JANUARY 11, 2025

The Data News are here to stay, the format might vary during the year, but here we are for another year. We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables. HNY 2025 ( credits ) Happy new year ✨ I wish you the best for 2025. Not really digest.

Data 130

Data Data Warehouse Programming Language BI 130

ProjectPro

JUNE 6, 2025

Becoming a Databricks Certified Data Engineer Associate is essential for data engineers as Databricks enables data engineers to efficiently process large volumes of data, build complex data pipelines, and leverage cloud-native services for enhanced reliability and cost-effectiveness.

Data Engineering 40

Data Engineering Data Engineer Engineering Certification 40

ProjectPro

JUNE 6, 2025

Welcome to our guide on How to Crack the Amazon Data Engineer Interview in 2024! million, Amazon heavily relies on data engineers for its success. With a 30% year-over-year increase in hiring data engineers, Amazon underscores its commitment to leveraging big data effectively.

Data Engineering 40

Data Engineering Data Engineer Engineering NoSQL 40

ProjectPro

JUNE 6, 2025

This comprehensive blog will help you discover how implementing some proven data engineering best practices can transform your workflow and tackle data engineering challenges. In the big data domain, every click, purchase, and interaction is valuable information.

Data Workflow 40

Data Workflow Data Engineering Data Engineer Data Cleanse 40

ProjectPro

JUNE 6, 2025

Azure Databricks embodies this philosophy by providing a user-friendly interface that simplifies data engineering complexities, helping professionals extract meaningful insights and drive business value. According to a report by IDC, worldwide data generation is projected to reach a staggering 175 zettabytes by 2025.

Data Engineer 40

Data Engineer Data Engineering Engineering Data Lake 40

ProjectPro

JUNE 6, 2025

With the proliferation of data sources, IoT devices, and edge nodes, almost 2.5 quintillion bytes of data is produced daily. This data is distributed across many platforms, including cloud databases, websites, CRM tools, social media channels, email marketing, etc. Table of Contents What is Azure Data Factory?

Data Engineer 40

Data Engineer Data Engineering Engineering Hospitality 40

ProjectPro

JUNE 6, 2025

Azure Data Factory and Databricks are two popular cloud-based data integration and ETL tools that can handle various types of data, including structured-unstructured data, and batch-streaming data. Here is a detailed comparison between Azure Databricks vs. Data Factory.

Data Engineer 40

Data Engineer Data Engineering Engineering Project 40

Start Data Engineering

JUNE 7, 2025

Performance depends on the execution engine 3. Introduction As a data engineer, CTEs are one of the best techniques you can use to improve query readability. Temp table enables you to reuse logic multiple times in a session 2.3. Conclusion 4. Recommended reading 1.

SQL 130

SQL Data Engineering Data Engineer Coding 130

ProjectPro

JUNE 6, 2025

These three database functions have been combined into a single tool to enable data extraction from one database and to store or maintain it in another. ETL is considered to be an essential part of data warehousing architecture in business processes worldwide. What SQL commands allow you to validate data completion?

Data Engineer 45

Data Engineer Data Engineering ETL Tools Engineering 45

Towards Data Science

JANUARY 30, 2025

Building more efficient AI TLDR : Data-centric AI can create more efficient and accurate models. I experimented with data pruning on MNIST to classify handwritten digits. What if I told you that using just 50% of your training data could achieve better results than using the fulldataset? Image byauthor.

Database-centric 106

Database-centric Datasets Data Architecture 106

ProjectPro

JUNE 6, 2025

This A-Z guide will walk you through the AWS Data Engineer Certification, providing insights, tips, and resources to streamline your certification journey. This AWS data engineer roadmap unfolds a step-by-step guide through the AWS Data Engineer Certification process.

AWS 40

AWS Certification Data Engineering Data Engineer 40

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262