Remove Data Engineer Remove Data Engineering Remove Data Science
article thumbnail

Data Engineering Weekly #181

Data Engineering Weekly

Editor’s Note: A New Series on Data Engineering Tools Evaluation There are plenty of data tools and vendors in the industry. Data Engineering Weekly is launching a new series on software evaluation focused on data engineering to better guide data engineering leaders in evaluating data tools.

article thumbnail

Data Engineering Weekly #182

Data Engineering Weekly

Switching from Apache Spark to Ray improves compact 12X larger datasets than Apache Spark, improves cost efficiency by 91%, and processes 13X more data per hour. What are the different roles inside the data engineering functions, and what should their ratio be?

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

The State of Data Engineering at Data Day Texas 2024

Jesse Anderson

The premier of my latest talk covering The State of Data Engineering. This starts with data warehousing and goes into data science. I finish off by showing how data engineering can avoid the same fate as data warehousing and data science.

article thumbnail

Mr. Pavan’s Data Engineering Journey Drives Business Success

Analytics Vidhya

He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Thus, providing valuable insights into the field of data engineering. Introduction We had an amazing opportunity to learn from Mr. Pavan.

article thumbnail

GPT and LLMs from a Data Engineering Perspective

Jesse Anderson

There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. How can we use LLMs in data engineering?

article thumbnail

Data Engineer vs Data Scientist: Which Career to Choose?

Analytics Vidhya

In the world of data, two crucial roles play a significant part in unlocking the power of information: Data Scientists and Data Engineers. But what sets these wizards of data apart? Welcome to the ultimate showdown of Data Scientist vs Data Engineer! appeared first on Analytics Vidhya.

article thumbnail

Brief History of Data Engineering

Jesse Anderson

As clearly evident as that sounds now, my writing about needing data engineering went heavily against the grain of everything that was written at the time. DJ Patil coined the term Data Scientist in 2008. For the majority of companies, that was the only title working on data problems at scale.