Data Engineering Podcast

JUNE 16, 2024

Summary Stripe is a company that relies on data to power their products and business. To support that functionality they have invested in Trino and Iceberg for their analytical workloads. In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform.

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Weekly

JUNE 16, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Learn More → Databricks: Open Sourcing Unity Catalog This week brought many exciting developments, with Snowflake and Databricks announcing open-source catalogs.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Cloudyard

JUNE 16, 2024

Read Time: 1 Minute, 37 Second In modern data-driven environments, it’s common to receive daily data feeds from various sources. These files need to be ingested into a data warehouse like Snowflake for further processing and analysis. Automating this process ensures data is consistently and reliably loaded without manual intervention. Suppose you are a data engineer at a company that receives daily sales data from an external vendor.

Cloud Storage 52

Cloud Storage Data Warehouse AWS Data Engineering 52