Analytics Vidhya

SEPTEMBER 12, 2024

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge.

Data Pipeline 144

Data Pipeline Building Management Data 144

Analytics Vidhya

NOVEMBER 8, 2023

In the data-driven world […] The post Monitoring Data Quality for Your Big Data Pipelines Made Easy appeared first on Analytics Vidhya. Determine success by the precision of your charts, the equipment’s dependability, and your crew’s expertise. A single mistake, glitch, or slip-up could endanger the trip.

Big Data 246

Big Data Data Pipeline Data IT 246

Ascend.io

NOVEMBER 14, 2024

Why Future-Proofing Your Data Pipelines Matters Data has become the backbone of decision-making in businesses across the globe. The ability to harness and analyze data effectively can make or break a company’s competitive edge. Resilience and adaptability are the cornerstones of a future-proof data pipeline.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Seattle Data Guy

APRIL 19, 2024

How data is structured, managed and processed will continue to grow in importance as the demand for AI and machine learning increase. It’s unavoidable that as businesses demand that their data teams implement AI, they will also realize that data engineers are a crucial piece of the data pipeline.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Pipeline 130

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data.

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Analytics Vidhya

FEBRUARY 19, 2023

Introduction Data pipelines play a critical role in the processing and management of data in modern organizations. A well-designed data pipeline can help organizations extract valuable insights from their data, automate tedious manual processes, and ensure the accuracy of data processing.

Data Pipeline 197

Data Pipeline Data Designing Data Process 197

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. Who are the data engineers?

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Confessions of a Data Guy

MARCH 29, 2024

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? The post Building Databricks Data Pipelines 101 appeared first on Confessions of a Data Guy. What does it look like, what tools do you use?

Data Pipeline 100

Data Pipeline Building Data IT 100

Confessions of a Data Guy

JANUARY 13, 2023

Rust has been on my mind a lot lately, probably because of Data Engineering boredom, watching Spark clusters chug along like some medieval farm worker endlessly trudging through the muck and mire of life. appeared first on Confessions of a Data Guy. appeared first on Confessions of a Data Guy.

Data Pipeline 147

Data Pipeline Data Engineer Data Engineering Data 147

Confessions of a Data Guy

SEPTEMBER 9, 2023

In the vast world of data, it’s not just about gathering and analyzing information anymore; it’s also about ensuring that data pipelines, processes, and platforms run seamlessly and efficiently.

Data Engineer 162

Data Engineer Data Engineering Engineering Data Pipeline 162

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure.

Data Engineer 103

Data Engineer Data Engineering Engineering Kafka 103

Data Engineering Weekly

JUNE 24, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. A few highlights from the report Unstructured data goes mainstream. AI-driven code development is going mainstream now.

Data Engineer 109

Data Engineer Data Engineering Engineering Google Cloud 109

Data Engineering Weekly

JUNE 30, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Looming regulatory requirements, data quality, governance issues & model accuracy keep failing enterprises.

Data Engineer 102

Data Engineer Data Engineering Engineering Unstructured Data 102

Start Data Engineering

MAY 13, 2021

What is an idempotent function Pre-requisites Why idempotency matters Making your data pipeline idempotent Conclusion Further reading References What is an idempotent function “Idempotence is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the (..)

Data Pipeline 148

Data Pipeline Computer Science Data Data Engineer 148

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineer 225

Data Engineer Data Engineering Python Engineering 225

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineer 105

Data Engineer Data Engineering Engineering Metadata 105

Data Engineering Weekly

NOVEMBER 11, 2024

To identify express queries, we developed a method using historical data to predict whether an upcoming query is an express query. The blog is an excellent overview of building dynamic data pipelines with Airflow datasets and pub/sub. Minimize execution failures due to executor OOM errors.

Data Engineer 57

Data Engineer Data Engineering Engineering Datasets 57

Data Engineering Weekly

AUGUST 25, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70

Data Engineering Weekly

SEPTEMBER 2, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. Try For Free → Conference Alert: Data Engineering for AI/ML This is a virtual conference at the intersection of Data and AI.

Data Engineer 67

Data Engineer Data Engineering Engineering Entertainment 67

Data Engineering Weekly

NOVEMBER 3, 2024

The challenges around memory, data size, and runtime are exciting to read. Sampling is an obvious strategy for data size, but the layered approach and dynamic inclusion of dependencies are some key techniques I learned with the case study. This count helps to ensure data consistency when deleting and compacting segments.

Data Engineer 62

Data Engineer Data Engineering Engineering Pipeline-centric 62

Netflix Tech

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Data Engineering Weekly

SEPTEMBER 22, 2024

Editor’s Note: Coming Next on Comparison Matrix Series - Data LakeHouse Our mission is to empower data professionals and organizations to make informed, data-driven decisions by providing a comprehensive buyer's guide and comparison matrix for selecting the best data tools.

Data Engineer 70

Data Engineer Data Engineering Engineering Data Lake 70

Start Data Engineering

FEBRUARY 22, 2024

Data Pipeline Logging Best Practices 3.1. Metadata: Information about pipeline runs, & data flowing through your pipeline 3.2. Introduction 2. Setup & Logging architecture 3. Obtain visibility into the code’s execution sequence using text logs 3.3. Understand resource usage by tracking Metrics 3.4.

Metadata 130

Metadata Data Engineer Data Engineering Engineering 130

Cloudera

SEPTEMBER 17, 2020

For enterprise organizations, managing and operationalizing increasingly complex data across the business has presented a significant challenge for staying competitive in analytic and data science driven markets. CDP data lifecycle integration and SDX security and governance. Enterprise Data Engineering From the Ground Up.

Data Pipeline 104

Data Pipeline Data Engineer Data Engineering Engineering 104

Data Engineering Weekly

AUGUST 4, 2024

Take Astro (the fully managed Airflow solution) for a test drive today and unlock a suite of features designed to simplify, optimize, and scale your data pipelines. As the next edition of the evaluation series, we are looking to take Data Observability. The blog makes an excellent point on what makes data illiterate.

Data Engineer 80

Data Engineer Data Engineering Engineering Data 80

Monte Carlo

OCTOBER 31, 2024

The rise of AI and GenAI has brought about the rise of new questions in the data ecosystem – and new roles. One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations.

Data Engineer 52

Data Engineer Data Engineering Engineering Unstructured Data 52

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. What is feature engineering is and why/to whom it matters?

Machine Learning 130

Machine Learning Data Engineer Data Engineering Engineering 130

Ascend.io

OCTOBER 8, 2024

Data engineering is the backbone of any data-driven organization, responsible for building and maintaining the infrastructure that supports data collection, storage, and analysis. Traditionally, data engineers have focused on the technical aspects of data management, ensuring data pipelines run smoothly and efficiently.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Architecture 52

Seattle Data Guy

FEBRUARY 11, 2023

Apache Airflow is a very popular tool that data engineers rely on. Why do data engineers like Airflow? What are… Read more The post What Is Apache Airflow – Data Engineering Consulting appeared first on Seattle Data Guy. Also, what does Apache Airflow event do? What is a DAG?

Consulting 147

Consulting Data Engineer Data Engineering Engineering 147

KDnuggets

AUGUST 28, 2024

Learn how to build and deploy an end-to-end data pipeline using Prefect with a few lines of code.

Data Pipeline 80

Data Pipeline Building Data Coding 80

Data Engineering Weekly

OCTOBER 20, 2024

Notion: A brief history of Notion’s data catalog Notion writes about its journey in adopting data catalogs and describes how a vanilla data catalog solution will only be effective if it adopts a strong data platform foundation. Many companies would have done something similar, so I’m glad it is finally coming.

Data Engineer 52

Data Engineer Data Engineering Engineering Manufacturing 52

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineer 189

Data Engineer Data Engineering Engineering Project 189

Start Data Engineering

OCTOBER 12, 2021

Introduction Responsibilities of a data engineer 1. Move data between systems 2. Manage data warehouse 3. Schedule, execute, and monitor data pipelines 4. Serve data to the end-users 5. Data strategy for the company 6.

Data Engineer 130

Data Engineer Data Engineering Engineering Data Warehouse 130

Start Data Engineering

DECEMBER 12, 2021

Introduction If you are building data pipelines from the ground up, the number of available data engineering tools to choose from can be overwhelming. Components 4. Choosing tools 4.1 Requirement x Component framework 4.2 Conclusion 6. Further reading 1.

Data Pipeline 130

Data Pipeline Data Engineer Data Engineering Data 130

Data Engineering Weekly

SEPTEMBER 15, 2024

Uber: DataMesh - How Uber laid the foundations for the Data Lake Cloud migration Many companies are slowly adopting DataMesh, and Uber writes about adopting the data mesh principle. Whether or not Data Mesh is a separate product is debatable, but it is certainly an impactful framework for scaling data platforms.

Data Engineer 67

Data Engineer Data Engineering Engineering Data Lake 67

Data Engineering Weekly

OCTOBER 6, 2024

Regularly review if pipelines are still required. Minimize the data used in pipelines, aka do incremental data pipeline design. Optimize pipeline schedules. Filter data effectively to make sure the query uses partition pruning. Don’t query two years of data. Data quality rules, that is.

Data Engineer 62

Data Engineer Data Engineering Engineering Data Warehouse 62