Sun.Dec 03, 2023

article thumbnail

Designing Data Transfer Systems That Scale

Data Engineering Podcast

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Andrei Tserakhau has dedicated his careeer to this problem, and in this episode he shares the lessons that he has learned and the work he is doing on his most recent data transfer system at DoubleCloud.

Systems 130
article thumbnail

Data Engineering Weekly #151

Data Engineering Weekly

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Visit rudderstack.com to learn more. Github: The architecture of today’s LLM applications LLM is slowly changing the application architecture landscape as it becomes integral to app development.

article thumbnail

Data Modelling For Data Engineers

Towards Data Science

The definitive guide for beginners Continue reading on Towards Data Science »