The Pragmatic Engineer

JANUARY 21, 2025

Hi, this is Gergely with a bonus issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. Fresh data shows how bad things are, courtesy of software engineer, Theodore R.

Software Engineer 328

Software Engineer Software Engineering Engineering Coding 328

The Pragmatic Engineer

MARCH 12, 2024

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. They come up with test cases and data. The tester. Also responsible for scaffolding of tests. 

Engineering 318

Engineering Programming Language Hospitality Insurance 318

Data Engineering Weekly

MAY 25, 2025

Dagster Components is now here Components provides a modular architecture that enables data practitioners to self-serve while maintaining engineering quality. The blog is an excellent compilation of types of query engines on top of the lakehouse, its internal architecture, and benchmarking against various categories.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Netflix Tech

DECEMBER 17, 2024

This article is the first in a multi-part series sharing a breadth of Analytics Engineering work at Netflix, recently presented as part of our annual internal Analytics Engineering conference. Subsequent posts will detail examples of exciting analytic engineering domain applications and aspects of the technical craft.

Engineering 90

Engineering Entertainment Amazon Web Services Utilities 90

Data Engineering Weekly

JUNE 1, 2025

Join Dagster and Neurospace to learn: - How to build AI pipelines with orchestration baked in - How to track data lineage for audits and traceability - Tips for designing compliant workflows under the EU AI Act Register for the technical session DuckDB: DuckLake - SQL as a Lakehouse Format DuckDB announced a new open table format, DuckLake.

Data Engineering 70

Data Engineering Data Engineer Engineering Relational Database 70

Christophe Blefari

JANUARY 11, 2025

The Data News are here to stay, the format might vary during the year, but here we are for another year. We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables. A few things to notice: Interest in AI grew by 190%, Prompt Engineering by 456%.

Data 130

Data Data Warehouse Programming Language BI 130

ProjectPro

JUNE 6, 2025

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. use SQL, compared to 61.7%

Data Engineering 45

Data Engineering Data Engineer SQL Engineering 45

ProjectPro

JUNE 6, 2025

In the thought process of making a career transition from ETL developer to data engineer job roles? Read this blog to know how various data-specific roles, such as data engineer, data scientist, etc., Data analysis and visualization have traditionally been a common goal for businesses.

Data Engineering 59

Data Engineering Data Engineer Engineering ETL Tools 59

ProjectPro

JUNE 6, 2025

Data engineering is the foundation for data science and analytics by integrating in-depth knowledge of data technology, reliable data governance and security, and a solid grasp of data processing. Data engineers need to meet various requirements to build data pipelines.

AWS 52

AWS Data Engineering Data Engineer Engineering 52

ProjectPro

JUNE 6, 2025

If you are planning to make a career transition into data engineering and want to know how to become a data engineer, this is the perfect place to begin your journey. Beginners will especially find it helpful if they want to know how to become a data engineer from scratch. in the following few sections.

Data Engineering 58

Data Engineering Data Engineer Engineering Amazon Web Services 58

databricks

JUNE 11, 2025

Events Data + AI Summit Data + AI World Tour Data Intelligence Days Event Calendar Blog and Podcasts Databricks Blog Explore news, product announcements, and more Databricks Mosaic Research Blog Discover the latest in our Gen AI research Data Brew Podcast Let’s talk data! REGISTER Ready to get started?

Entertainment 132

Entertainment Manufacturing Consulting Retail 132

ProjectPro

JUNE 6, 2025

The demand for data-related roles has increased massively in the past few years. Companies are actively seeking talent in these areas, and there is a huge market for individuals who can manipulate data, work with large databases and build machine learning algorithms. What is an AI Engineer? What does an AI Engineer do?

Engineering 75

Engineering Software Engineer Software Engineering Deep Learning 75

ProjectPro

JUNE 6, 2025

Most of us have observed that data scientist is usually labeled the hottest job of the 21st century, but is it the only most desirable job? No, that is not the only job in the data world. These trends underscore the growing demand and significance of data engineering in driving innovation across industries.

Data Engineering 59

Data Engineering Data Engineer Project Engineering 59

Data Engineering Weekly

DECEMBER 28, 2024

In this post, we delve into predictions for 2025, focusing on the transformative role of AI agents, workforce dynamics, and data platforms. For professionals across domains—data engineers, AI engineers, and data scientists—the message is clear: adapt or become obsolete.

Unstructured Data 112

Unstructured Data Metadata Government Data 112

ProjectPro

JUNE 6, 2025

Over the past few years, data-related jobs have drastically increased. Previously, the spotlight was on gaining relevant insights from data, but recently, data handling has gained attention. Because of that, data engineer jobs have garnered recognition and popularity.

Data Engineering 49

Data Engineering Data Engineer Engineering Hadoop 49

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 116

Data Warehouse SQL Programming Language Data 116

ProjectPro

JUNE 6, 2025

The total amount of data that was created in 2020 was 64 zettabytes! The volume and the variety of data captured have also rapidly increased, with critical system sources such as smartphones, power grids, stock exchanges, and healthcare adding more data sources as the storage capacity increases.

Data Ingestion 45

Data Ingestion Data Engineering Data Engineer Project 45

ProjectPro

JUNE 6, 2025

This blog post provides an overview of the top 10 data engineering tools for building a robust data architecture to support smooth business operations. Table of Contents What are Data Engineering Tools? Dice Tech Jobs report 2020 indicates Data Engineering is one of the highest in-demand jobs worldwide.

Data Engineering 45

Data Engineering Data Engineer Engineering Kafka 45

ProjectPro

JUNE 6, 2025

Data engineering has become crucial to any modern organization's technology stack. The need for fast and efficient data processing is high, as companies increasingly rely on data to make business decisions and improve product quality. But what books should you read if you want to learn more about data engineering?

Data Engineering 45

Data Engineering Data Engineer Engineering Lambda Architecture 45

ProjectPro

JUNE 6, 2025

Becoming a data engineer can be challenging, but we are here to make the journey easier. In this blog, we have curated a list of the best data engineering courses so you can master this challenging field with confidence. Say goodbye to confusion and hello to a clear path to data engineering expertise!

Data Engineering 45

Data Engineering Data Engineer Cloud Engineering 45

Towards Data Science

JANUARY 30, 2025

Building more efficient AI TLDR : Data-centric AI can create more efficient and accurate models. I experimented with data pruning on MNIST to classify handwritten digits. What if I told you that using just 50% of your training data could achieve better results than using the fulldataset? Image byauthor.

Database-centric 106

Database-centric Datasets Data Architecture 106

Seattle Data Guy

MARCH 31, 2025

RevOps teams want to streamline processes… Read more The post Best Automation Tools In 2025 for Data Pipelines, Integrations, and More appeared first on Seattle Data Guy. But automation isnt just for analytics.

Data Pipeline 195

Data Pipeline Machine Learning Data Process 195

ProjectPro

JUNE 6, 2025

With over 175 full features service offerings, organizations are head hunting for AWS data engineers who can help them build and maintain the entire AWS cloud infrastructure to keep the applications up and running. Cloud platforms are becoming the new standard for managing an organization's data.

AWS 45

AWS Data Engineering Data Engineer Amazon Web Services 45

Start Data Engineering

JUNE 7, 2025

Performance depends on the execution engine 3. Introduction As a data engineer, CTEs are one of the best techniques you can use to improve query readability. CTEs make medium-complex SQL easy to understand 2.2. Temp table enables you to reuse logic multiple times in a session 2.3. Conclusion 4. Recommended reading 1.

SQL 130

SQL Data Engineering Data Engineer Coding 130

Towards Data Science

DECEMBER 10, 2024

Three Zero-Cost Solutions That Take Hours, NotMonths A data quality certified pipeline. Source: unsplash.com In my career, data quality initiatives have usually meant big changes. Whats more, fixing the data quality issues this way often leads to new problems. Create a custom dashboard for your specific data qualityproblem.

PostgreSQL 75

PostgreSQL Data SQL Python 75

Seattle Data Guy

APRIL 15, 2025

Planning out your data infrastructure in 2025 can feel wildly different than it did even five years ago. Everyone is talking about AI, chatbots, LLMs, vector databases, and whether your data stack is “AI-ready.” The ecosystem is louder, flashier, and more fragmented.

Database 182

Database Data IT Big Data 182

DataKitchen

JUNE 6, 2025

Data Quality Testing: A Shared Resource for Modern Data Teams In today’s AI-driven landscape, where data is king, every role in the modern data and analytics ecosystem shares one fundamental responsibility: ensuring that incorrect data never reaches business customers. Each role touches data differently.

Data Ingestion 73

Data Ingestion Government Data Governance Data 73

Simon Späti

MARCH 28, 2025

The database landscape has reached 394 ranked systems across multiple categoriesrelational, document, key-value, graph, search engine, time series, and the rapidly emerging vector databases. As AI applications multiply quickly, vector technologies have become a frontier that data engineers must explore.

Technology 130

Technology PostgreSQL MySQL Database 130

Snowflake

JUNE 10, 2025

Data transformations are the engine room of modern data operations — powering innovations in AI, analytics and applications. As the core building blocks of any effective data strategy, these transformations are crucial for constructing robust and scalable data pipelines.

Data Pipeline 83

Data Pipeline SQL Python Building 83

ProjectPro

JUNE 6, 2025

These three database functions have been combined into a single tool to enable data extraction from one database and to store or maintain it in another. ETL is considered to be an essential part of data warehousing architecture in business processes worldwide. What SQL commands allow you to validate data completion?

Data Engineering 45

Data Engineering Data Engineer ETL Tools Data Mining 45