Analytics Vidhya

NOVEMBER 8, 2023

In the data-driven world […] The post Monitoring Data Quality for Your Big Data Pipelines Made Easy appeared first on Analytics Vidhya. Determine success by the precision of your charts, the equipment’s dependability, and your crew’s expertise. A single mistake, glitch, or slip-up could endanger the trip.

Big Data 246

Big Data Data Pipeline Data IT 246

Ascend.io

NOVEMBER 14, 2024

Why Future-Proofing Your Data Pipelines Matters Data has become the backbone of decision-making in businesses across the globe. The ability to harness and analyze data effectively can make or break a company’s competitive edge. Resilience and adaptability are the cornerstones of a future-proof data pipeline.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data.

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Analytics Vidhya

FEBRUARY 19, 2023

Introduction Data pipelines play a critical role in the processing and management of data in modern organizations. A well-designed data pipeline can help organizations extract valuable insights from their data, automate tedious manual processes, and ensure the accuracy of data processing.

Data Pipeline 197

Data Pipeline Data Designing Data Process 197

Start Data Engineering

APRIL 8, 2021

Understand the current state of the data pipeline 1. Timelines & deliverables Deliver iteratively Conclusion Further reading References Introduction As data engineers, you will have to re-engineer legacy data pipelines. While re-engineering data pipelines, if you have struggled with

Data Pipeline 130

Data Pipeline Engineering Data Engineering Data Engineer 130

Confessions of a Data Guy

MARCH 29, 2024

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? The post Building Databricks Data Pipelines 101 appeared first on Confessions of a Data Guy. What does it look like, what tools do you use?

Data Pipeline 100

Data Pipeline Building Data IT 100

Analytics Vidhya

JUNE 25, 2023

In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.

Data Engineering 220

Data Engineering Data Engineer Engineering Data Pipeline 220

Start Data Engineering

DECEMBER 11, 2022

Data pipeline patterns 3.1. Multi-hop pipelines 3.3.2. Conditional/ Dynamic pipelines 3.3.3. Disconnected data pipelines 4. Source Ordering 2.3. Sink Overwritability 3. Extraction patterns 3.1.1. Time ranged 3.1.2. Full Snapshot 3.1.3. Lookback 3.1.4. Streaming 3.2. Behavioral 3.2.1. Idempotent 3.2.2.

Data Pipeline 196

Data Pipeline Designing Data 196

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. We’ll answer the question, “What are data pipelines?” Table of Contents What are Data Pipelines?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Confessions of a Data Guy

JANUARY 13, 2023

Rust has been on my mind a lot lately, probably because of Data Engineering boredom, watching Spark clusters chug along like some medieval farm worker endlessly trudging through the muck and mire of life. appeared first on Confessions of a Data Guy.

Data Pipeline 147

Data Pipeline Data Engineering Data Engineer Data 147

LinkedIn Engineering

JUNE 26, 2023

However, we've found that this vertical self-service model doesn't work particularly well for data pipelines, which involve wiring together many different systems into end-to-end data flows. Data pipelines power foundational parts of LinkedIn's infrastructure, including replication between data centers.

Data Pipeline 112

Data Pipeline Kafka SQL MySQL 112

Striim

OCTOBER 11, 2024

A well-executed data pipeline can make or break your company’s ability to leverage real-time insights and stay competitive. Thriving in today’s world requires building modern data pipelines that make moving data and extracting valuable insights quick and simple. What is a Data Pipeline?

Data Pipeline 52

Data Pipeline MongoDB Unstructured Data Data Lake 52

Start Data Engineering

MAY 13, 2021

What is an idempotent function Pre-requisites Why idempotency matters Making your data pipeline idempotent Conclusion Further reading References What is an idempotent function “Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the (..)

Data Pipeline 148

Data Pipeline Computer Science Data Data Engineering 148

Analytics Vidhya

APRIL 3, 2023

Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers. Nevertheless, setting up a streaming data pipeline to power such dashboards may […] The post Data Engineering for Streaming Data on GCP appeared first on Analytics Vidhya.

Data Engineering 251

Data Engineering Data Engineer Engineering Data 251

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Netflix Tech

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Start Data Engineering

OCTOBER 12, 2021

Introduction Testing your data pipeline 1. Data quality testing 3. Unit and contract testing Conclusion Further reading Introduction Testing data pipelines are different from testing other applications, like a website backend. End-to-end system testing 2. Monitoring and alerting 4.

Data Pipeline 174

Data Pipeline Data Systems 174

Seattle Data Guy

APRIL 19, 2024

It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Pipeline 130

Precisely

DECEMBER 28, 2023

Business success is based on how we use continuously changing data. That’s where streaming data pipelines come into play. This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. What is a streaming data pipeline?

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Start Data Engineering

SEPTEMBER 16, 2021

Introduction Choosing tools/frameworks to scale your data pipelines can be confusing. If you have struggled with Data pipelines that randomly crash Finding guides on how to scale your data pipelines from the ground up Then this post is for you. Choose your scaling strategy 5. Conclusion 6. Further reading 7.

Data Pipeline 130

Data Pipeline Data IT 130

KDnuggets

AUGUST 28, 2024

Learn how to build and deploy an end-to-end data pipeline using Prefect with a few lines of code.

Data Pipeline 80

Data Pipeline Building Data Coding 80

Picnic Engineering

FEBRUARY 8, 2023

In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. In this post, we will explain how our team automates the creation of new data pipeline deployments. Now, we can have a pipeline ready in minutes.

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. Data stacks are becoming more and more complex.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Snowflake

JUNE 4, 2024

The rise of AI, for example, will depend on the collaboration between data and development. We’re increasingly seeing software engineering workloads that are deeply intertwined with a strong data foundation. Additionally, Dynamic Tables are a new table type that you can use at every stage of your processing pipeline.

Data Pipeline 102

Data Pipeline Python SQL Government 102

Data Engineering Podcast

JUNE 30, 2024

Petr shares his journey from being an engineer to founding Synq, emphasizing the importance of treating data systems with the same rigor as engineering systems. He discusses the challenges and solutions in data reliability, including the need for transparency and ownership in data systems.

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Start Data Engineering

DECEMBER 12, 2021

Introduction If you are building data pipelines from the ground up, the number of available data engineering tools to choose from can be overwhelming. Components 4. Choosing tools 4.1 Requirement x Component framework 4.2 Conclusion 6. Further reading 1.

Data Pipeline 130

Data Pipeline Data Engineering Data Engineer Data 130

Data Engineering Weekly

JUNE 24, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Bessemer publishes the Data + AI infrastructure market map to help companies understand the landscape and the key players.

Data Engineering 109

Data Engineering Data Engineer Engineering Google Cloud 109

Snowflake

OCTOBER 3, 2023

We are excited to announce the availability of data pipelines replication, which is now in public preview. In the event of an outage, this powerful new capability lets you easily replicate and failover your entire data ingestion and transformations pipelines in Snowflake with minimal downtime.

Data Pipeline 88

Data Pipeline Management Data Ingestion Data 88

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

SEPTEMBER 17, 2020

With growing disparate data across everything from edge devices to individual lines of business needing to be consolidated, curated, and delivered for downstream consumption, it’s no wonder that data engineering has become the most in-demand role across businesses — growing at an estimated rate of 50% year over year.

Data Pipeline 104

Data Pipeline Data Engineering Data Engineer Engineering 104

Hevo

SEPTEMBER 11, 2024

A Data Pipeline is an indispensable part of a data engineering workflow. It enables the extraction, transformation, and storage of data across disparate data sources and ensures that the right data is available at the right time.

Data Pipeline 40

Data Pipeline Python Coding Data Engineering 40

Towards Data Science

APRIL 3, 2023

Data pipeline management done right simplifies deployment and increases the availability and accessibility of data for analytics Continue reading on Towards Data Science »

Data Pipeline 76

Data Pipeline Data Science Data Accessible 76

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. As I have shared , its impact on data engineering is exciting.

Data Engineering 103

Data Engineering Data Engineer Engineering Kafka 103

Data Engineering Weekly

NOVEMBER 11, 2024

Slack: Empowering Engineers with AI In our most recent internal developer survey, 65% of respondents said they prefer an IDE as the primary platform to engage with generative-AI for development tasks. The blog is an excellent overview of building dynamic data pipelines with Airflow datasets and pub/sub.

Data Engineering 57

Data Engineering Data Engineer Engineering Datasets 57

Data Engineering Weekly

JUNE 30, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Engineering jobs saw a 10.42% decline. Learn More → Ozge Demirci, Jonas Hannane & Xinrong Zhu: Who Is AI Replacing?

Data Engineering 102

Data Engineering Data Engineer Engineering Unstructured Data 102

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.

Data Engineering 162

Data Engineering Data Engineer Engineering Data Pipeline 162

Ascend.io

APRIL 26, 2023

Data teams worldwide are building data pipelines with the point solutions that make up the “modern data stack.” Pipelines built with this approach are slow and require constant manual reprogramming and updates. For a deep dive into this level of automation, take a look at our What Is Data Pipeline Automation paper.

Data Pipeline 52

Data Pipeline Datasets Metadata Data 52