Towards Data Science

OCTOBER 20, 2024

Strategies for Enhancing Generalizability, Scalability, and Maintainability in Your ETL Pipelines Continue reading on Towards Data Science »

Python 75

Python Data Science Data Programming 75

Scott Logic

OCTOBER 20, 2024

Introduction The intersection of sustainability and software development is gaining attention, with increasing awareness of the environmental impact of IT practices. A recent blog by Pini Reznik discusses the environmental impact of Agile software development, highlighting how its emphasis on speed and adaptability often leads to resource waste, particularly in cloud computing.

Cloud Computing 52

Cloud Computing Cloud Coding Architecture 52

Data Engineering Weekly

OCTOBER 20, 2024

Notion: A brief history of Notion’s data catalog Notion writes about its journey in adopting data catalogs and describes how a vanilla data catalog solution will only be effective if it adopts a strong data platform foundation. Adopting Typescript rather than the specialized IDL languages is a good strategy, although I wonder how it works in cross-language systems like Android & iOS.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Scott Logic

OCTOBER 20, 2024

Introducing, the Multi-cloud Today most software projects utilise some form of cloud computing to host their infrastructure, often choosing one of the big three providers, Google (GCP), Microsoft (Azure) or Amazon (AWS). What I think is less common, is to consider multiple cloud providers when looking to host resources although it is something which is mentioned from time to time.

Cloud 52

Cloud AWS Cloud Computing Government 52

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline