Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure.

Data Engineer 103

Data Engineer Data Engineering Engineering Kafka 103

Data Engineering Weekly

AUGUST 25, 2024

Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI. Instead, you will see engineers, product managers, and founders talking about the data engineering practices of AI/ML. It is not a conference for the hype.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70

Data Engineering Podcast

JUNE 4, 2023

Summary A significant portion of the time spent by data engineering teams is on managing the workflows and operations of their pipelines. Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Lake 130

Data Engineering Weekly

OCTOBER 6, 2024

Join us on October 10th at 9am PT | 12pm ET as I walk through how data observability platforms like Monte Carlo can help data analytics teams reduce time spent on data quality by 80% or more. Though the blog talks about Superset, every team should strive to adopt a similar framework to manage the lifecycle of their dashboards.

Data Engineer 62

Data Engineer Data Engineering Engineering Data Warehouse 62

Start Data Engineering

MAY 11, 2022

Data lake structure 5. Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Prerequisite 4.2 AWS Infrastructure costs 4.3 Code walkthrough 5.1 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7. Next steps 9.

Data Engineer 246

Data Engineer Data Engineering Project Data Lake 246

Netflix Tech

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineer 81

Data Engineer Data Engineering Engineering Data Warehouse 81

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineer 189

Data Engineer Data Engineering Engineering Project 189

Data Engineering Weekly

SEPTEMBER 2, 2024

Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI. Instead, you will see engineers, product managers, and founders talking about the data engineering practices of AI/ML. It is not a conference for the hype.

Data Engineer 67

Data Engineer Data Engineering Engineering Entertainment 67

Data Engineering Weekly

SEPTEMBER 22, 2024

I’m fairly confident that a framework that supports a deterministic way of building a data pipeline and a conversational way of building business logic is the future of data pipeline engineering. Dropbox writes a case study on integrating Lakera Guard into their prompt engineering to secure the LLM infrastructure.

Data Engineer 70

Data Engineer Data Engineering Engineering Data Lake 70

Start Data Engineering

OCTOBER 12, 2021

Introduction Responsibilities of a data engineer 1. Move data between systems 2. Manage data warehouse 3. Schedule, execute, and monitor data pipelines 4. Serve data to the end-users 5. Data strategy for the company 6.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Warehouse 130

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. You’re left wondering if there’s a breaking point where your DIY data solution won’t cut it anymore—and honestly, you might be there already. It means you’re scaling!

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Data Engineering Weekly

MAY 26, 2024

[link] Meta: Composable data management at Meta Meta writes about its transition to a composable data management system to improve interoperability, reusability, and engineering efficiency. It is a long standing question on people wondering In what situations should you use SQL instead of Pandas as a data scientist?

Data Engineer 76

Data Engineer Data Engineering Engineering Google Cloud 76

Towards Data Science

JULY 26, 2024

Learn from Software Engineers and Discover the Joy of ‘Worse is Better’ Thinking source: unsplash.com Recently, I have had the fortune of speaking to a number of data engineers and data architects about the problems they face with data in their businesses. That was it. So, What the Heck is Radical Simplicity?

Data Engineer 57

Data Engineer Data Engineering Engineering Data Warehouse 57

Knowledge Hut

JULY 5, 2024

Large enterprises heavily rely on data for informed decision-making, and this reliance is where data engineers step in. Data engineers like myself play a pivotal role in assessing infrastructure and taking relevant actions. Looking ahead, the future of data engineering appears promising.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Cleanse 52

Analytics Vidhya

FEBRUARY 21, 2023

This is where data warehousing is a critical component of any business, allowing companies to store and manage vast amounts of data. It provides the necessary foundation for businesses to […] The post Understanding the Basics of Data Warehouse and its Structure appeared first on Analytics Vidhya.

Data Warehouse 206

Data Warehouse IT Data Collection Data 206

Analytics Vidhya

FEBRUARY 2, 2023

Now, businesses are looking for different types of data storage to store and manage their data effectively. Organizations can collect millions of data, but if they’re lacking in storing that data, those efforts […] The post A Comprehensive Guide to Data Lake vs. Data Warehouse appeared first on Analytics Vidhya.

Data Lake 202

Data Lake Data Warehouse Data Storage Data 202

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Confessions of a Data Guy

OCTOBER 7, 2024

Has the Data Warehouse finally died, has that unruly upstart the Lake House finally taken its place atop the seething mass of data we call home? Can we say that after all these decades the Data Warehouse Toolkit […] The post The Death of the Data Warehouse, replaced by the Lake House.

Data Warehouse 113

Data Warehouse IT Data Big Data 113

Towards Data Science

DECEMBER 4, 2023

A Glossary with Use Cases for First-Timers in Data Engineering An happy Data Engineer at work Are you a data engineering rookie interested in knowing more about modern data infrastructures? In this guide Data Engineering meets Formula 1. Data models are built around business needs.

Data Engineer 81

Data Engineer Data Engineering Engineering Data Lake 81

Data Engineering Podcast

MARCH 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin. As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20.

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Data Engineering Weekly

JULY 7, 2024

The dataset organization as a set of LSM trees seems an interesting approach, which sounds similar to Google’s Napa data warehouse system. The author highlights Paimon’s consistency model by examining the metadata model.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineer 130

Data Engineer Data Engineering Engineering Project 130

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. As a result, the 2020 election was the first I can remember where the on-call engineers weren’t on the edge of their seats, white-knuckling their keyboards…At least not for system reliability reasons.

Data Engineer 95

Data Engineer Data Engineering Engineering Software Engineer 95

Data Engineering Weekly

APRIL 14, 2024

link] Github: 4 ways GitHub engineers use GitHub Copilot The impact of LLM on software development is undeniable. Github shares some insights on how Github engineers use Github Copilot. It can drastically reduce the computational cost and energy requirement for training LLM, and it is an interesting development to watch.

Data Engineer 62

Data Engineer Data Engineering Engineering Business Intelligence 62

Seattle Data Guy

MAY 26, 2024

Many data engineers and analysts don’t realize how valuable the knowledge they have is. They’ve spent hours upon hours learning SQL, Python, how to properly analyze data, build data warehouses, and understand the differences between eight different ETL solutions.

Consulting 130

Consulting Engineering Data Warehouse SQL 130

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability. To measure, but not track.

Data Pipeline 98

Data Pipeline Data Engineer Data Engineering Engineering 98

Data Engineering Weekly

FEBRUARY 18, 2024

Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. We are so over the Big Data Era to Modern Data Stack. Elevate your data skills!

Data Engineer 62

Data Engineer Data Engineering Engineering Data 62

Maxime Beauchemin

AUGUST 28, 2017

This post follows up on The Rise of the Data Engineer , a recent post that was an attempt at defining data engineering and described how this new role relates to historical and modern roles in the data space. Understanding and exposing the adversity that the role is facing is a first step towards finding solutions.

Data Engineer 174

Data Engineer Data Engineering Engineering Software Engineer 174

Hepta Analytics

FEBRUARY 24, 2022

Examples of such systems are: ATM Shopping cart MoMo ticket systems OLAP on the other hand is used for data analysis. Data is aggregated from multiple sources and used by the business team to make decisions. Data structure and schema is defined in advance. The main goal is to become the single source of truth in an organization.

Data Warehouse 52

Data Warehouse Data Engineer Data Engineering Engineering 52

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineer 78

Data Engineer Data Engineering Engineering SQL 78

U-Next

MARCH 1, 2023

Introduction Data Engineer is responsible for managing the flow of data to be used to make better business decisions. A solid understanding of relational databases and SQL language is a must-have skill, as an ability to manipulate large amounts of data effectively. In 2022, data engineering will hold a share of 29.8%

Data Engineer 75

Data Engineer Data Engineering Non-relational Database Engineering 75

Data Science Blog: Data Engineering

DECEMBER 23, 2022

Warum Data Engineering der Data Science in Bedeutung und Berufschancen längst die Show stiehlt, dabei selbst ebenso einem stetigen Wandel unterliegt. Was ein Data Engineer wirklich können muss Der Data Scientist als sexiest Job des 21. Data Engineer, der gefragteste Job des 21sten Jahrhunderts?

Data Engineer 52

Data Engineer Data Engineering Engineering Business Intelligence 52

Acceldata

FEBRUARY 16, 2023

Why Data Engineers Must Manage Data Pipelines, Not Data Warehouses

Data Warehouse 52

Data Warehouse Data Pipeline Data Engineer Data Engineering 52

Knowledge Hut

OCTOBER 31, 2023

Two of the most well-known cloud service providers, Amazon Web Services (AWS) and Microsoft Azure, provide reliable data engineering solutions. Often, aspiring data engineers must choose between two options: AWS data engineer or Azure data engineer.

Data Engineer 52

Data Engineer Data Engineering AWS Engineering 52

Data Engineering Weekly

SEPTEMBER 8, 2024

How We Created Data Science Personas for Spotify’s Analytics Platform It takes a village to engineer data. It requires balanced teamwork with multiple organizational personalities to cultivate a data-driven culture. Spotify writes an excellent article highlighting the key Data Science persona.

Data Engineer 52

Data Engineer Data Engineering Engineering Kafka 52

Seattle Data Guy

FEBRUARY 1, 2023

Recently several consulting calls started with people asking, “Do we need a data warehouse?” ” This isn’t a question about whether you need data warehouse consultants, but instead whether you should event start a data warehouse project. Not every company needs a data warehouse.

Data Warehouse 130

Data Warehouse Consulting Data Project 130

Towards Data Science

JULY 30, 2024

Rethinking data warehousing: Why redefinition is necessary even beyond Modern Data Warehouse (MDW) and Lakehouse Models Continue reading on Towards Data Science »

Data Warehouse 56

Data Warehouse Data Science Data Data Architecture 56