Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services and systems need to be kept up to date, but so does the code that controls their behavior. In this episode your host Tobias Macey reflects on his current challenges in this area and some of the factors that contribute to the complexity of the problem.

Management 147

Management Data Lake High Quality Data Machine Learning 147

databricks

MAY 12, 2024

How we reached 79.9% on the Spider dev dataset with Llama3 8B through savvy prompting and fine-tuning on Databricks.

Datasets 130

Datasets 130

Scott Logic

MAY 12, 2024

While Large Language Models excel at generating answers fast, the latest models are not yet skilled at performing very specific tasks that require more ‘System 2’ thinking , the more analytical kind that relies on reflection and self-critique. This is where agents are changing the AI game , bringing action and direction, like a rounded team of highly skilled colleagues, experts in their own field doing what each does best to fulfil a user’s goal or intention.

Python 52

Python Coding Architecture Accessible 52

databricks

MAY 12, 2024

How we reached 79.9% on the Spider dev dataset with Llama3 8B through savvy prompting and fine-tuning on Databricks.

Datasets 80

Datasets 80

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Data Engineering Weekly

MAY 12, 2024

Editor’s Note: DEWCon Call for Speakers Open - September 13th, Bengaluru - India DEWCon is back this year on a grand scale on September 13th, 2024 , in Bengaluru, India. This year, we added some additional features to bring the data community together. Book a 1:1 session with experts on career, tech stack, team management, and more!! " Ideas Jam Session ,” where you can talk about your idea/ prototypes in a 10-minute slot More details on DEWCon will be in the coming weeks, and we wil

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52