Confessions of a Data Guy

JULY 24, 2024

We all keep hearing about Arrow this and Arrow that … seems every new tool built today for Data Engineering seems to be at least partly based on Arrow’s in-memory format. So, […] The post PyArrow vs Polars (vs DuckDB) for Data Pipelines. appeared first on Confessions of a Data Guy.

Data Pipeline 162

Data Pipeline Data Engineer Data Engineering Data 162

Start Data Engineering

DECEMBER 11, 2022

Data pipeline patterns 3.1. Multi-hop pipelines 3.3.2. Conditional/ Dynamic pipelines 3.3.3. Disconnected data pipelines 4. Source Ordering 2.3. Sink Overwritability 3. Extraction patterns 3.1.1. Time ranged 3.1.2. Full Snapshot 3.1.3. Lookback 3.1.4. Streaming 3.2. Behavioral 3.2.1. Idempotent 3.2.2.

Data Pipeline 195

Data Pipeline Designing Data 195

KDnuggets

NOVEMBER 25, 2024

Building data pipelines is a very important skill that you should learn as a data engineer. A data pipeline is just a series of procedures that transport data from one location to another, frequently changing it along the way.

Data Pipeline 131

Data Pipeline Building Transportation Data 131

Data Engineering Podcast

JUNE 25, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Engineer 225

Data Engineer Data Engineering Python Engineering 225

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineer 189

Data Engineer Data Engineering Engineering Project 189

Start Data Engineering

MAY 13, 2021

What is an idempotent function Pre-requisites Why idempotency matters Making your data pipeline idempotent Conclusion Further reading References What is an idempotent function “Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the (..)

Data Pipeline 147

Data Pipeline Computer Science Data Data Engineer 147

Snowflake

APRIL 17, 2024

This traditional SQL-centric approach often challenged data engineers working in a Python environment, requiring context-switching and limiting the full potential of Python’s rich libraries and frameworks. To get started, explore the comprehensive API documentation , which will guide you through every step.

Data Pipeline 113

Data Pipeline Python Data Engineer Data Engineering 113

Start Data Engineering

OCTOBER 12, 2021

Introduction Testing your data pipeline 1. Data quality testing 3. Unit and contract testing Conclusion Further reading Introduction Testing data pipelines are different from testing other applications, like a website backend. End-to-end system testing 2. Monitoring and alerting 4.

Data Pipeline 173

Data Pipeline Data Systems 173

KDnuggets

MARCH 17, 2020

As the role of the data engineer continues to grow in the field of data science, so are the many tools being developed to support wrangling all that data. Five of these tools are reviewed here (along with a few bonus tools) that you should pay attention to for your data pipeline work.

Data Engineer 159

Data Engineer Data Engineering Engineering Project 159

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability.

Data Pipeline 98

Data Pipeline Data Engineer Data Engineering Engineering 98

Confessions of a Data Guy

JANUARY 13, 2023

Rust has been on my mind a lot lately, probably because of Data Engineering boredom, watching Spark clusters chug along like some medieval farm worker endlessly trudging through the muck and mire of life. appeared first on Confessions of a Data Guy.

Data Pipeline 147

Data Pipeline Data Engineer Data Engineering Data 147

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data.

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Netflix Tech

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.

Data Engineer 162

Data Engineer Data Engineering Engineering Data Pipeline 162

Analytics Vidhya

SEPTEMBER 12, 2024

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge.

Data Pipeline 144

Data Pipeline Building Management Data 144

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Seattle Data Guy

APRIL 19, 2024

It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Pipeline 130

KDnuggets

AUGUST 28, 2024

Learn how to build and deploy an end-to-end data pipeline using Prefect with a few lines of code.

Data Pipeline 120

Data Pipeline Building Data Coding 120

Start Data Engineering

DECEMBER 12, 2021

Introduction If you are building data pipelines from the ground up, the number of available data engineering tools to choose from can be overwhelming. Components 4. Choosing tools 4.1 Requirement x Component framework 4.2 Conclusion 6. Further reading 1.

Data Pipeline 130

Data Pipeline Data Engineer Data Engineering Data 130

Start Data Engineering

SEPTEMBER 16, 2021

Introduction Choosing tools/frameworks to scale your data pipelines can be confusing. If you have struggled with Data pipelines that randomly crash Finding guides on how to scale your data pipelines from the ground up Then this post is for you. Choose your scaling strategy 5. Conclusion 6. Further reading 7.

Data Pipeline 130

Data Pipeline Data IT 130

Start Data Engineering

OCTOBER 12, 2021

Introduction Responsibilities of a data engineer 1. Move data between systems 2. Manage data warehouse 3. Schedule, execute, and monitor data pipelines 4. Serve data to the end-users 5. Data strategy for the company 6.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Warehouse 130

Data Engineering Podcast

FEBRUARY 18, 2024

In this episode Dain Sundstrom, CTO of Starburst, explains how the combination of the Trino query engine and the Iceberg table format offer the ease of use and execution speed of data warehouses with the infinite storage and scalability of data lakes. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

MAY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex. Data lakes are notoriously complex. My thanks to the team at Code Comments for their support.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Data Engineering Weekly

JUNE 24, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Should we term this as an AI product curve of disappointment ?

Data Engineer 111

Data Engineer Data Engineering Engineering Google Cloud 111

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. Data stacks are becoming more and more complex.

Data Pipeline 130

Data Pipeline Building MongoDB MySQL 130

Start Data Engineering

OCTOBER 11, 2021

Data modeling 4.1 Data warehousing 4.2 Data pipelines 6. Probabilistic data structures (optional) Interview prep, the TL;DR version Conclusion Introduction Are you a student, analyst, engineer, or someone preparing for a data engineering interview and overwhelmed by all the tools and concepts?

Data Engineer 130

Data Engineer Data Engineering Engineering Cloud Computing 130

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineer 104

Data Engineer Data Engineering Engineering Kafka 104

KDnuggets

SEPTEMBER 5, 2023

Build a streaming data pipeline using Formula 1 data, Python, Kafka, RisingWave as the streaming database, and visualize all the real-time data in Grafana.

Data Pipeline 146

Data Pipeline Kafka Building Python 146

Start Data Engineering

OCTOBER 22, 2021

Data pipeline 2.4. Data analytics 3. Introduction SQL is the bread and butter of data engineering. Narrow transformations 2.2.1.2. Wide transformations 2.2.2. Query planner 2.2.3. Security & Permissions 2.3. Practice 4. Conclusion 5. Further reading 6. References 1.

SQL 130

SQL Data Engineer Data Engineering Engineering 130

Start Data Engineering

NOVEMBER 11, 2021

Writing an end-to-end data pipeline test 4. Introduction Data pipelines can have multiple software components. If you are wondering What is the best way to end-to-end test data pipelines? In this post, we go over some techniques that you can use to set up end-to-end tests for your data pipelines.

Data Pipeline 130

Data Pipeline Cloud Data 130

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineer Data Engineering Engineering 147

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineer 106

Data Engineer Data Engineering Engineering Metadata 106

databricks

JUNE 13, 2024

Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines.

Data Engineer 144

Data Engineer Data Engineering Engineering Data Pipeline 144

Confessions of a Data Guy

MARCH 29, 2024

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? The post Building Databricks Data Pipelines 101 appeared first on Confessions of a Data Guy. What does it look like, what tools do you use?

Data Pipeline 100

Data Pipeline Building Data IT 100

Data Engineering Weekly

JUNE 30, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. It is a good reminder to the data industry that we need to solve the fundamentals of data engineering to utilize AI better.

Data Engineer 103

Data Engineer Data Engineering Engineering Unstructured Data 103

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. How is this distinct from other forms of data pipeline development and delivery?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Start Data Engineering

FEBRUARY 22, 2024

Data Pipeline Logging Best Practices 3.1. Metadata: Information about pipeline runs, & data flowing through your pipeline 3.2. Introduction 2. Setup & Logging architecture 3. Obtain visibility into the code’s execution sequence using text logs 3.3. Understand resource usage by tracking Metrics 3.4.

Metadata 130

Metadata Data Engineer Data Engineering Engineering 130