ETL Pipelines in Python: Best Practices and Techniques
Towards Data Science
OCTOBER 20, 2024
Strategies for Enhancing Generalizability, Scalability, and Maintainability in Your ETL Pipelines Continue reading on Towards Data Science »
Sun.Oct 20, 2024
57 
Sustainability in Agile: How Scrum Roles Can Drive Greener Practices by James Camilleri
Scott Logic
OCTOBER 20, 2024
Introduction The intersection of sustainability and software development is gaining attention, with increasing awareness of the environmental impact of IT practices. A recent blog by Pini Reznik discusses the environmental impact of Agile software development, highlighting how its emphasis on speed and adaptability often leads to resource waste, particularly in cloud computing.
Data Engineering Weekly #194
Data Engineering Weekly
OCTOBER 20, 2024
Notion: A brief history of Notion’s data catalog Notion writes about its journey in adopting data catalogs and describes how a vanilla data catalog solution will only be effective if it adopts a strong data platform foundation. Adopting Typescript rather than the specialized IDL languages is a good strategy, although I wonder how it works in cross-language systems like Android & iOS.
Into The Multi-cloud by Dave Ogle
Scott Logic
OCTOBER 20, 2024
Introducing, the Multi-cloud Today most software projects utilise some form of cloud computing to host their infrastructure, often choosing one of the big three providers, Google (GCP), Microsoft (Azure) or Amazon (AWS). What I think is less common, is to consider multiple cloud providers when looking to host resources although it is something which is mentioned from time to time.
Apache Airflow® Best Practices for ETL and ELT Pipelines
Advertisement
Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.
Let's personalize your content