Jesse Anderson

SEPTEMBER 14, 2023

There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. How can we use LLMs in data engineering?

Data Engineer 147

Data Engineer Data Engineering Engineering NoSQL 147

Seattle Data Guy

APRIL 19, 2024

It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Pipeline 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Go to [dataengineeringpodcast.com/materialize]([link] Support Data Engineering Podcast Summary This podcast started almost exactly six years ago, and the technology landscape was much different than it is now.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

Seattle Data Guy

JANUARY 17, 2023

Are data engineers all learning Rust? Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy. Are Snowflake and Databricks still fighting over total cost of ownership?

Data Engineer 211

Data Engineer Data Engineering Engineering Data 211

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineer Data Engineering Engineering 147

Data Engineering Weekly

AUGUST 4, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. The Archmage thrift proxy layer to hide the infrastructure and enable easier version upgrades is a good engineering practice that I would recommend for any platform team.

Data Engineer 80

Data Engineer Data Engineering Engineering Data 80

Seattle Data Guy

MAY 20, 2023

Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools.

Data Engineer 147

Data Engineer Data Engineering Project Engineering 147

Knowledge Hut

JULY 5, 2024

Large enterprises heavily rely on data for informed decision-making, and this reliance is where data engineers step in. Data engineers like myself play a pivotal role in assessing infrastructure and taking relevant actions. Looking ahead, the future of data engineering appears promising.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Cleanse 52

Analytics Vidhya

JANUARY 31, 2023

Introduction Big data is revolutionizing the healthcare industry and changing how we think about patient care. In this case, big data refers to the vast amounts of data generated by healthcare systems and patients, including electronic health records, claims data, and patient-generated data.

Healthcare 251

Healthcare Big Data Electronics Data 251

Analytics Vidhya

NOVEMBER 8, 2023

In the data-driven world […] The post Monitoring Data Quality for Your Big Data Pipelines Made Easy appeared first on Analytics Vidhya. Determine success by the precision of your charts, the equipment’s dependability, and your crew’s expertise. A single mistake, glitch, or slip-up could endanger the trip.

Big Data 246

Big Data Data Pipeline Data IT 246

Analytics Vidhya

FEBRUARY 23, 2023

Introduction Big Data is a large and complex dataset generated by various sources and grows exponentially. It is so extensive and diverse that traditional data processing methods cannot handle it. The volume, velocity, and variety of Big Data can make it difficult to process and analyze.

Big Data Tools 234

Big Data Tools Big Data Datasets Data 234

Confessions of a Data Guy

NOVEMBER 5, 2022

There are probably few things in life that will strike more fear and tumult in the heart of the Data Engineer than historical loads. How could it possibly be, just take a bunch of data stored somewhere and shove it into a table. […] The post Introduction to Historical Loads – for Data Engineers.

Data Engineer 130

Data Engineer Data Engineering Engineering Data 130

Team Data Science

JANUARY 8, 2021

Big Data has become the dominant innovation in all high-performing companies. Notable businesses today focus their decision-making capabilities on knowledge gained from the study of big data. Big Data gives you an advantage in competition as true for businesses as it is for professionals working in the area of analytics.

Data Engineer 130

Data Engineer Data Engineering Engineering Unstructured Data 130

Netflix Tech

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineer 81

Data Engineer Data Engineering Engineering Data Warehouse 81

Data Engineering Weekly

FEBRUARY 18, 2024

Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. We are so over the Big Data Era to Modern Data Stack. Elevate your data skills!

Data Engineer 62

Data Engineer Data Engineering Engineering Data 62

Monte Carlo

OCTOBER 31, 2024

One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations. But what does an AI data engineer do? Table of Contents What Does an AI Data Engineer Do?

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Knowledge Hut

MARCH 7, 2024

Foresighted enterprises are the ones who will be able to leverage this data for maximum profitability through data processing and handling techniques. With the rise in opportunities related to Big Data, challenges are also bound to increase. Below are the 5 major Big Data challenges that enterprises face in 2024: 1.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineer 98

Data Engineer Data Engineering Coding Project 98

Towards Data Science

JULY 26, 2024

Learn from Software Engineers and Discover the Joy of ‘Worse is Better’ Thinking source: unsplash.com Recently, I have had the fortune of speaking to a number of data engineers and data architects about the problems they face with data in their businesses. So, What the Heck is Radical Simplicity?

Data Engineer 57

Data Engineer Data Engineering Engineering Data Warehouse 57

WeCloudData

JULY 17, 2024

Data engineering is a hot topic in recent years, mainly due to the rise of artificial intelligence, big data, and data science. For enterprises, data is full of infinite value. Every enterprise is transforming in the direction of digitalization.

Data Engineer 52

Data Engineer Data Engineering Engineering Big Data 52

Analytics Vidhya

FEBRUARY 5, 2023

Introduction In this technical era, Big Data is proven as revolutionary as it is growing unexpectedly. According to the survey reports, around 90% of the present data was generated only in the past two years. Big data is nothing but the vast volume of datasets measured in terabytes or petabytes or even more.

Big Data 205

Big Data Hadoop Datasets Data 205

Data Engineering Weekly

MAY 12, 2024

[link] Gradient Flow: Learning from the Past - Comparing the Hype Cycles of Big Data and GenAI The blogs compare the hype cycle of Big Data with Gen AI. The blog narrates the initial hype of Big Data, followed by talent shortages and lead time to build production applications.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Lake 52

Striim

OCTOBER 4, 2024

Join us as we sit down with Joe Reis, live at Big Data LDN (London) 2024. and AWS through his new course on Data Engineering. Joe’s new course promises to elevate your data skills with hands-on exercises that marry foundational knowledge with cutting-edge practices.

Big Data 52

Big Data AWS Data Architecture Certification 52

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. Why Python for Data Engineering?

Data Engineer 52

Data Engineer Data Engineering Python Engineering 52

Data Engineering Weekly

MARCH 3, 2024

Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering. Part 1: Why did we need to build our own SIEM?

Data Engineer 67

Data Engineer Data Engineering Engineering Pipeline-centric 67

Edureka

AUGUST 29, 2024

Data engineering is a core branch pertaining to big data and analytics — an area focusing on the creation of scalable and optimizable pipelines for data. The average salary for a data engineer foreshadows a good compensation package since they play a critical role in handling and optimizing large data sets.

Data Engineer 40

Data Engineer Data Engineering Engineering Portfolio 40

Data Engineering Weekly

DECEMBER 24, 2023

I love the rising, stable, and declining format for categorizing data engineering trends. My take on the rising trends 🟡 Intelligent Data Platform: Yes, I fully agree. However, I’m less optimistic about the “multi-engine” orchestrator part. The technique for geomasking is an excellent read.

Data Engineer 62

Data Engineer Data Engineering Engineering Deep Learning 62

Emeritus

NOVEMBER 8, 2022

As you read this, people across the world are texting, posting on social media, and searching on Google, adding to the growing volume of big data. And as big data’s quantity increases so does its significance for companies. Big data has become a pivotal resource to generate information and make insightful decisions.

Big Data 52

Big Data Data Engineer Data Engineering Engineering 52

KDnuggets

JANUARY 7, 2020

An estimated 8,650% growth of the volume of Data to 175 zetabytes from 2010 to 2025 has created an enormous need for Data Engineers to build an organization's big data platform to be fast, efficient and scalable.

Data Engineer 155

Data Engineer Data Engineering Engineering Big Data 155

Maxime Beauchemin

AUGUST 28, 2017

This post follows up on The Rise of the Data Engineer , a recent post that was an attempt at defining data engineering and described how this new role relates to historical and modern roles in the data space. In my experience, it’s rare to find any sort of decent dev or test environments in the big data world.

Data Engineer 174

Data Engineer Data Engineering Engineering Software Engineer 174

ProjectPro

SEPTEMBER 26, 2021

Big Data Engineer is one of the most popular job profiles in the data industry. This blog on Big Data Engineer salary gives you a clear picture of the salary range according to skills, countries, industries, job titles, etc. Big Data gets over 1.2 What does a big data engineer do?

Big Data 52

Big Data Data Engineer Data Engineering Engineering 52

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It provides high-throughput access to data and is optimized for […] The post A Dive into the Basics of Big Data Storage with HDFS appeared first on Analytics Vidhya.

Data Storage 185

Data Storage Big Data Hadoop Datasets 185

WeCloudData

JUNE 21, 2024

Data engineering is a hot topic in recent years, mainly due to the rise of artificial intelligence, big data, and data science. For enterprises, data is full of infinite value. Every enterprise is transforming in the direction of digitalization.

Data Engineer 52

Data Engineer Data Engineering Engineering Big Data 52

Data Engineering Weekly

MARCH 10, 2024

Intuit advocate starting with existing models and enhancing them through prompt engineering or training on domain-specific knowledge before considering custom LLM development. link] Richard Crowley: SOC 2 compliance for startups and first-timers While SOC 2 might seem outside a data engineer's wheelhouse, understanding its principles is key.

Data Engineer 52

Data Engineer Data Engineering Engineering Datasets 52

Knowledge Hut

APRIL 25, 2024

Big data in information technology is used to improve operations, provide better customer service, develop customized marketing campaigns, and take other actions to increase revenue and profits. It is especially true in the world of big data. It is especially true in the world of big data.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Edureka

AUGUST 29, 2024

The responsibilities of a data engineer imply that the person in this role designs, creates, develops, and maintains systems and architecture that allow them to collect, store, and interpret data. What Does a Data Engineer Do? Why Choose Data Engineering as a Career? How to Become a Data Engineer?

Data Engineer 40

Data Engineer Data Engineering Engineering Generalist 40

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineer Data Engineering Cloud 130