Data Engineering Weekly

SEPTEMBER 29, 2024

Airbnb: Sandcastle - data/AI apps for everyone Product ideas powered by data and AI must go through rapid iteration on shareable, lightweight live prototypes instead of static proposals. However, hosting an internal application for fast prototyping is always a challenging platform to build and maintain. Airbnb writes about Sandcastle, an Airbnb-internal prototyping platform that enables data scientists, engineers, and product managers to bring data/AI ideas to life.

Data Engineer 64

Data Engineer Data Engineering Engineering SQL 64

dbt Developer Hub

SEPTEMBER 29, 2024

One of the most important things that dbt does is unlock the ability for teams to collaborate on creating and disseminating organizational knowledge. In the past, this primarily looked like a team working in one dbt Project to create a set of transformed objects in their data platform. As dbt was adopted by larger organizations and began to drive workloads at a global scale, it became clear that we needed mechanisms to allow teams to operate independently from each other, creating and sharing da

Metadata 52

Metadata Cloud Project Accessibility 52

Hevo

SEPTEMBER 29, 2024

Have you ever opened the billing section of a BigQuery account and got a shocking surprise? You are not alone. BigQuery is a powerful tool, but this power does not come for free all the time. It can quickly deplete your budget if you do not practice good cost management.

Management 52

Management IT Data 52

Hevo

SEPTEMBER 29, 2024

Today most organizations are of the opinion that public APIs should be tapped into and useful information extracted there from. The same, however triggers a sound ETL solution to handle the data correctly.

ETL Tools 52

ETL Tools Data Integration Data 52

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Hevo

SEPTEMBER 29, 2024

If you are a data-driven business, then you must know how crucial it is to extract meaningful insights from your data. That’s where Reverse ETL comes into play. I’m guessing you might know what ETL (Extract, Transform, Load) is. It is the process of bringing data into your warehouses.

ETL Tools 52

ETL Tools Process IT Data 52