Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services and systems need to be kept up to date, but so does the code that controls their behavior. In this episode your host Tobias Macey reflects on his current challenges in this area and some of the factors that contribute to the complexity of the problem.

Management 147

Management Data Lake High Quality Data Machine Learning 147

databricks

MAY 12, 2024

How we reached 79.9% on the Spider dev dataset with Llama3 8B through savvy prompting and fine-tuning on Databricks.

Datasets 126

Datasets 126

Scott Logic

MAY 12, 2024

While Large Language Models excel at generating answers fast, the latest models are not yet skilled at performing very specific tasks that require more ‘System 2’ thinking , the more analytical kind that relies on reflection and self-critique. This is where agents are changing the AI game , bringing action and direction, like a rounded team of highly skilled colleagues, experts in their own field doing what each does best to fulfil a user’s goal or intention.

Python 52

Python Coding Architecture Accessible 52

databricks

MAY 12, 2024

How we reached 79.9% on the Spider dev dataset with Llama3 8B through savvy prompting and fine-tuning on Databricks.

Datasets 80

Datasets 80

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Data Engineering Weekly

MAY 12, 2024

Editor’s Note: DEWCon Call for Speakers Open - September 13th, Bengaluru - India DEWCon is back this year on a grand scale on September 13th, 2024 , in Bengaluru, India. This year, we added some additional features to bring the data community together. Book a 1:1 session with experts on career, tech stack, team management, and more!! " Ideas Jam Session ,” where you can talk about your idea/ prototypes in a 10-minute slot More details on DEWCon will be in the coming weeks, and we wil

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52