Analytics Vidhya

FEBRUARY 19, 2023

Introduction Data pipelines play a critical role in the processing and management of data in modern organizations. A well-designed data pipeline can help organizations extract valuable insights from their data, automate tedious manual processes, and ensure the accuracy of data processing.

Data Pipeline 199

Data Pipeline Data Designing Data Process 199

Start Data Engineering

DECEMBER 11, 2022

Data pipeline patterns 3.1. Multi-hop pipelines 3.3.2. Conditional/ Dynamic pipelines 3.3.3. Disconnected data pipelines 4. Source Ordering 2.3. Sink Overwritability 3. Extraction patterns 3.1.1. Time ranged 3.1.2. Full Snapshot 3.1.3. Lookback 3.1.4. Streaming 3.2. Behavioral 3.2.1. Idempotent 3.2.2.

Data Pipeline 195

Data Pipeline Designing Data 195

Ascend.io

NOVEMBER 14, 2024

Why Future-Proofing Your Data Pipelines Matters Data has become the backbone of decision-making in businesses across the globe. The ability to harness and analyze data effectively can make or break a company’s competitive edge. Resilience and adaptability are the cornerstones of a future-proof data pipeline.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Start Data Engineering

OCTOBER 12, 2021

Introduction Testing your data pipeline 1. Data quality testing 3. Unit and contract testing Conclusion Further reading Introduction Testing data pipelines are different from testing other applications, like a website backend. End-to-end system testing 2. Monitoring and alerting 4.

Data Pipeline 173

Data Pipeline Data Systems 173

Confessions of a Data Guy

JULY 24, 2024

We all keep hearing about Arrow this and Arrow that … seems every new tool built today for Data Engineering seems to be at least partly based on Arrow’s in-memory format. So, […] The post PyArrow vs Polars (vs DuckDB) for Data Pipelines. appeared first on Confessions of a Data Guy.

Data Pipeline 162

Data Pipeline Data Engineering Data Engineer Data 162

Analytics Vidhya

SEPTEMBER 12, 2024

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge.

Data Pipeline 147

Data Pipeline Building Management Data 147

Netflix Tech

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Monte Carlo

NOVEMBER 21, 2024

Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way. That’s where data pipeline design patterns come in. Data Mesh Pattern 8.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

KDnuggets

NOVEMBER 25, 2024

Building data pipelines is a very important skill that you should learn as a data engineer. A data pipeline is just a series of procedures that transport data from one location to another, frequently changing it along the way.

Data Pipeline 122

Data Pipeline Building Transportation Data 122

KDnuggets

NOVEMBER 2, 2023

For production grade LLM apps, you need a robust data pipeline. This article talks about the different stages of building a Gen AI data pipeline and what is included in these stages.

Data Pipeline 147

Data Pipeline Building Data 147

Confessions of a Data Guy

JANUARY 13, 2023

Rust has been on my mind a lot lately, probably because of Data Engineering boredom, watching Spark clusters chug along like some medieval farm worker endlessly trudging through the muck and mire of life. appeared first on Confessions of a Data Guy.

Data Pipeline 147

Data Pipeline Data Engineering Data Engineer Data 147

KDnuggets

AUGUST 28, 2024

Learn how to build and deploy an end-to-end data pipeline using Prefect with a few lines of code.

Data Pipeline 110

Data Pipeline Building Data Coding 110

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data.

Data Pipeline 130

Data Pipeline BI SQL Machine Learning 130

KDnuggets

SEPTEMBER 5, 2023

Build a streaming data pipeline using Formula 1 data, Python, Kafka, RisingWave as the streaming database, and visualize all the real-time data in Grafana.

Data Pipeline 139

Data Pipeline Kafka Building Python 139

Snowflake

OCTOBER 3, 2023

We are excited to announce the availability of data pipelines replication, which is now in public preview. In the event of an outage, this powerful new capability lets you easily replicate and failover your entire data ingestion and transformations pipelines in Snowflake with minimal downtime.

Data Pipeline 114

Data Pipeline Management Data Ingestion Data 114

Confessions of a Data Guy

MARCH 29, 2024

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? The post Building Databricks Data Pipelines 101 appeared first on Confessions of a Data Guy. What does it look like, what tools do you use?

Data Pipeline 100

Data Pipeline Building Data IT 100

Hevo

DECEMBER 20, 2024

In this fast-paced digital era, multiple sources like IoT devices, social media platforms, and financial systems generate the data continuously and in real-time.    Every business wants to analyze these data in real-time to be ahead in the competitive game.  It has the ability to […]

Data Pipeline 40

Data Pipeline Media Data Systems 40

Snowflake

JUNE 4, 2024

Snowflake’s new Python API (GA soon) simplifies data pipelines and is readily available through pip install snowflake. Additionally, Dynamic Tables are a new table type that you can use at every stage of your processing pipeline. Interact with Snowflake objects directly in Python. Automate or code, the choice is yours.

Data Pipeline 122

Data Pipeline Python SQL Database 122

Snowflake

APRIL 17, 2024

Since the previous Python connector API mostly communicated via SQL, it also hindered the ability to manage Snowflake objects natively in Python, restricting data pipeline efficiency and the ability to complete complex tasks. Or, experience these features firsthand at our free Dev Day event on June 6th in the Demo Zone.

Data Pipeline 124

Data Pipeline Python Data Engineering Data Engineer 124

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. We’ll answer the question, “What are data pipelines?” Table of Contents What are Data Pipelines?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Start Data Engineering

APRIL 14, 2025

API Data extraction = GET-ting data from a server 3.1. GET data 3.1.1. GET data for a specific entity 3. Introduction 2. APIs are a way to communicate between systems on the Internet 2.1. HTTP is a protocol commonly used for websites 2.1.1. Request: Ask the Internet exactly what you want 2.1.2.

Data Pipeline 130

Data Pipeline Python Data Systems 130

Start Data Engineering

JANUARY 12, 2023

Introduction Sample project Code design patterns 1. Functional design 2. Factory pattern 3. Strategy pattern 4. Singleton, & Object pool patterns Python helpers 1. Dataclass 3. Context Managers 4. Testing with pytest 5.

Designing 147

Designing Coding Python Data Pipeline 147

Hevo

JANUARY 26, 2025

Is your business incapacitated due to slow and unreliable data pipelines in today’s hyper-competitive environment? Data pipelines are the backbone that guarantees real-time access to critical information for informed and quicker decisions. The data pipeline market is set to grow from USD 6.81

Data Pipeline 40

Data Pipeline Data Accessible Accessibility 40

databricks

JULY 8, 2024

We are proud to announce two new analyst reports recognizing Databricks in the data engineering and data streaming space: IDC MarketScape: Worldwide Analytic.

Data Pipeline 137

Data Pipeline Process Cloud Data Engineer 137

Simon Späti

JUNE 20, 2022

Data consumers, such as data analysts, and business users, care mostly about the production of data assets. On the other hand, data engineers have historically focused on modeling the dependencies between tasks (instead of data assets) with an orchestrator tool. How can we reconcile both worlds?

Data Pipeline 147

Data Pipeline Data Data Engineer Data Engineering 147

Start Data Engineering

FEBRUARY 10, 2025

Data transformations as functions lead to maintainable code 3. Track pipeline progress (logging, Observer) with objects 3.3. Use objects to store configurations of data systems (e.g., Class lets you define reusable code and pipeline patterns 4.1. Introduction 2. Objects help track things (aka state) 3.1. Spark, etc.)

Data Pipeline 130

Data Pipeline Python Coding Data 130

Cloudera

DECEMBER 4, 2024

Our customers rely on NiFi as well as the associated sub-projects (Apache MiNiFi and Registry) to connect to structured, unstructured, and multi-modal data from a variety of data sources – from edge devices to SaaS tools to server logs and change data capture streams. Cloudera DataFlow 2.9

Data Pipeline 93

Data Pipeline Data Ingestion Data Preparation Architecture 93

Start Data Engineering

FEBRUARY 3, 2025

Introduction If you’ve been in the data space long enough, you would have come across really long SQL scripts that someone had written years ago. Introduction 2. Split your SQL into smaller parts 2.1. Start with a baseline validation to ensure that your changes do not change the output too much 2.2. Conclusion 4. Required reading 1.

SQL 147

SQL Data Pipeline Data 147

Start Data Engineering

JUNE 24, 2024

Features crucial to building and maintaining data pipelines 2.1. Schedulers to run data pipelines at specified frequency 2.2. Orchestrators to define the order of execution of your pipeline tasks 2.2.1. Define the order of execution of pipeline tasks with a DAG 2.2.2. Introduction 2.

Data Pipeline 208

Data Pipeline Coding Building Data 208

Start Data Engineering

JUNE 14, 2024

Run Data Pipelines 2.1. Batch pipelines 3.3. Stream pipelines 3.4. Event-driven pipelines 3.5. LLM RAG pipelines 4. Introduction Whether you are new to data engineering or have been in the data field for a few years, one of the most challenging parts of learning new frameworks is setting them up!

Data Engineering 239

Data Engineering Data Engineer Project Engineering 239

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineering 258

Data Engineering Data Engineer Engineering Data 258

databricks

OCTOBER 24, 2023

"I can't think of anything that's been more powerful since the desktop computer." — Michael Carbin, Associate Professor, MIT, and Founding Advisor, MosaicML A.

Data Pipeline 119

Data Pipeline Building Data 119

Confessions of a Data Guy

JANUARY 8, 2025

You know, for all the hoards of content, books, and videos produced in the “Data Space” over the last few years, famous or others, it seems I find there are volumes of information on the pieces and parts of working in Data. appeared first on Confessions of a Data Guy.

Data Storage 124

Data Storage Data Pipeline Data IT 124

Analytics Vidhya

JUNE 25, 2023

A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines. While data scientists and analysts receive […] The post What Data Engineers Really Do? appeared first on Analytics Vidhya.

Data Engineering 226

Data Engineering Data Engineer Engineering Data Pipeline 226

Snowflake

NOVEMBER 11, 2024

Here’s how Snowflake Cortex AI and Snowflake ML are accelerating the delivery of trusted AI solutions for the most critical generative AI applications: Natural language processing (NLP) for data pipelines: Large language models (LLMs) have a transformative potential, but they often batch inference integration into pipelines, which can be cumbersome.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Analytics Vidhya

AUGUST 7, 2024

Introduction Apache Airflow is a crucial component in data orchestration and is known for its capability to handle intricate workflows and automate data pipelines. Many organizations have chosen it due to its flexibility and strong scheduling capabilities.

Data Pipeline 192

Data Pipeline Data Process Data Workflow 192