Data Engineering Podcast

JUNE 2, 2024

Summary Modern businesses aspire to be data driven, and technologists enjoy working through the challenge of building data systems to support that goal. Data governance is the binding force between these two parts of the organization. Nicola Askham found her way into data governance by accident, and stayed because of the benefit that she was able to provide by serving as a bridge between the technology and business.

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

databricks

JUNE 2, 2024

We are excited to announce a new data type called variant for semi-structured data. Variant provides an order of magnitude performance improvements compared.

Structured Data 136

Structured Data Data Data Engineer Data Engineering 136

Data Engineering Weekly

JUNE 2, 2024

Data Engineering Weekly is sponsored by Astronomer—Enterprise-Grade Apache Airflow. Deliver data on time with the speed and scale your application demands. Learn More → AI Verify Foundation: Model AI Governance Framework for Generative AI Several countries are working on building governance rules for Gen AI. Data sovereignty will play a vital role as countries formulate regulations.

Data Engineer 103

Data Engineer Data Engineering Engineering Pipeline-centric 103

Scott Logic

JUNE 2, 2024

It’s June! It’s Pride month! Rainbows! Love is love! We’re your ally! Buy stuff with rainbows on! Let’s come to your Pride parade, but make sure you tone it down a bit! More rainbows! Buy our products! Look, we’ve put a rainbow on it! We love everyone ! We love absolutely everyone, in a very non-specific way! We definitely love sparkly unicorn rainbows!

Consulting 98

Consulting AWS IT Management 98

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Zalando Engineering

JUNE 2, 2024

Introduction Our Data Science colleagues had been hosting an internal Data Science Days event for a few years. For our 2,000+ Engineers, we had been missing a similar community event. For several years we wanted to organize one, but got distracted by other priorities and external factors. Finally, in 2022 we decided to commit to hosting an internal Engineering Conference every year and included this commitment in our Engineering Strategy.

Engineering 63

Engineering Programming Data Science Data Engineering 63

Monte Carlo

JUNE 2, 2024

Every day, an uncountable amount of data flows through millions of businesses. Data Infrastructure Engineers are the professionals who ensure that this data flows smoothly and reliably. But what does a data infrastructure engineer do exactly? Table of Contents What is a Data Infrastructure Engineer? Difference Between a Data Infrastructure Engineer and a Data Science Engineer Skill Sets for Data Infrastructure Engineers Technical Expertise Day-to-Day Operations Collaboration with Cross-Functiona

Engineering 52

Engineering Amazon Web Services Data Science AWS 52