François Nguyen

FEBRUARY 28, 2021

Today, we are going to apply these principles to the data pipelines. The idea is to transpose these 7 principles to data pipeline knowing that Data pipelines are 100% flexible : if you have the skills, you build the pipeline you want. How does a bad data pipeline process look like ?

Data Pipeline 130

Data Pipeline Building Manufacturing BI 130

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Ascend.io

APRIL 26, 2023

Data teams worldwide are building data pipelines with the point solutions that make up the “modern data stack.” Pipelines built with this approach are slow and require constant manual reprogramming and updates. For a deep dive into this level of automation, take a look at our What Is Data Pipeline Automation paper.

Data Pipeline 52

Data Pipeline Datasets Metadata Data 52

Data Science Blog: Data Engineering

MAY 20, 2024

Continuous Integration and Continuous Delivery (CI/CD) for Data Pipelines: It is a Game-Changer with AnalyticsCreator! The need for efficient and reliable data pipelines is paramount in data science and data engineering. They transform data into a consistent format for users to consume.

Data Pipeline 52

Data Pipeline BI Data Lake Data Warehouse 52

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. How is the governance of DataHub being managed?

Metadata 100

Metadata BI Data Warehouse Government 100

Cloudera

MARCH 14, 2023

In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development. Key requirements for building data pipelines Every data pipeline starts with a business requirement.

Data Pipeline 82

Data Pipeline Designing Kafka Metadata 82

Ascend.io

MARCH 17, 2023

These engineering functions are almost exclusively concerned with data pipelines, spanning ingestion, transformation, orchestration, and observation — all the way to data product delivery to the business tools and downstream applications. Pipelines need to grow faster than the cost to run them.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineering 52

Ascend.io

MARCH 17, 2023

These engineering functions are almost exclusively concerned with data pipelines, spanning ingestion, transformation, orchestration, and observation — all the way to data product delivery to the business tools and downstream applications. Pipelines need to grow faster than the cost to run them.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineering 52

Ascend.io

SEPTEMBER 25, 2023

European data teams can now build and automate data pipelines with the ease of a managed SaaS offering while meeting stringent data sovereignty requirements. As Ascend’s partners Snowflake and Databricks continue to expand in the region, the demand for scalable data pipeline solutions has skyrocketed.

Data Pipeline 52

Data Pipeline Data Architect Metadata Media 52

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights.

Data Pipeline 62

Data Pipeline Data Raw Data Database 62

Ascend.io

NOVEMBER 28, 2023

The terms ‘data orchestration’ and ‘data pipeline orchestration’ are often used interchangeably, yet they diverge significantly in function and scope. Data orchestration refers to a wide collection of methods and tools that coordinate any and all types of data-related computing tasks.

Data Pipeline 52

Data Pipeline IT Data Data Engineer 52

Christophe Blefari

MARCH 15, 2024

Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. Forward thinking Dataviz is hierarchical — Malloy, once again, provides an excellent article about a new way to see data visualisations. This is Croissant. It's inspirational.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Ascend.io

MAY 8, 2023

The costs of developing and running data pipelines are coming under increasing scrutiny because the bills for infrastructure and data engineering talent are piling up. For data teams, it is time to ask: “How can we have an impact on these runaway costs and still deliver unprecedented business value?”

Data Pipeline 52

Data Pipeline Data Ingestion Metadata Data 52

Ascend.io

FEBRUARY 2, 2023

2, 2023 — Ascend.io, the leader in Data Pipeline Automation for building the world’s most intelligent data pipelines, is expanding its global presence and has hired Robin Rostratter as Head of EMEA sales to better engage and support the company’s customer base across Europe. . SAN FRANCISCO , Feb.

Data Pipeline 52

Data Pipeline Google Cloud Metadata Media 52

Ascend.io

FEBRUARY 2, 2023

2, 2023 — Ascend.io, the leader in Data Pipeline Automation for building the world’s most intelligent data pipelines, is expanding its global presence and has hired Robin Rostratter as Head of EMEA sales to better engage and support the company’s customer base across Europe. . SAN FRANCISCO , Feb.

Data Pipeline 52

Data Pipeline Google Cloud Metadata Media 52

Cloudera

FEBRUARY 8, 2021

Below is the entire set of steps in the data lifecycle, and each step in the lifecycle will be supported by a dedicated blog post(see Fig. 1): Data Collection – data ingestion and monitoring at the edge (whether the edge be industrial sensors or people in a vehicle showroom).

Data Pipeline 116

Data Pipeline Building Manufacturing Data Warehouse 116

Data Engineering Podcast

JUNE 16, 2024

what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? What were the requirements and selection criteria that led to the selection of that combination of technologies?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

Data Engineering Podcast

AUGUST 13, 2022

Data stacks are becoming more and more complex. This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating the quality of the data and causing teams to lose trust. Data stacks are becoming more and more complex.

Metadata 100

Metadata MongoDB Scala MySQL 100

Netflix Tech

NOVEMBER 14, 2023

We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Furthermore, we’ll delve into the inner workings of Psyberg, its unique features, and how it integrates into our data pipelining workflows. This is mainly used to identify new changes since the last update.

Data Engineer 107

Data Engineer Data Engineering Engineering Metadata 107

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Towards Data Science

OCTOBER 25, 2023

As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. You probably work closely with the colleague who builds the data pipeline that you were consuming. There was only one data team, two at most. null, null).

Data Pipeline 78

Data Pipeline SQL Data Architecture Data 78

Towards Data Science

APRIL 6, 2023

Today’s post follows the same philosophy: fitting local and cloud pieces together to build a data pipeline. And, when it comes to data engineering solutions, it’s no different: They have databases, ETL tools, streaming platforms, and so on — a set of tools that makes our life easier (as long as you pay for them). not sponsored.

Data Pipeline 75

Data Pipeline AWS Amazon Web Services Python 75

dbt Developer Hub

SEPTEMBER 14, 2021

Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! They'll share how to build a scalable data pipeline using dbt + Snowflake for financial reporting use cases. Things to Observe ? This photo of me on top of Mt Whitney after my solo SOBO of the JMT last month.

Metadata 52

Metadata Data Pipeline Python Cloud 52

Netflix Tech

NOVEMBER 14, 2023

Now, let’s explore the state of our pipelines after incorporating Psyberg. Pipelines After Psyberg Let’s explore how different modes of Psyberg could help with a multistep data pipeline. The session metadata table can then be read to determine the pipeline input.

Metadata 99

Metadata Data Pipeline Scala Data Workflow 99

Data Engineering Podcast

NOVEMBER 20, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

DECEMBER 18, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Struggling with broken pipelines? Missing data? Atlan is the metadata hub for your data ecosystem.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Ascend.io

APRIL 12, 2023

From this research, we developed a framework with a sequence of stages to implement data integrity quickly and measurably via data pipelines. Table of Contents Why does data integrity matter? At every level of a business, individuals must trust the data, so they can confidently make timely decisions. Let’s explore!

Data Pipeline 52

Data Pipeline Data Integration Datasets Data 52

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 29, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Struggling with broken pipelines? Missing data? Atlan is the metadata hub for your data ecosystem.

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

NOVEMBER 13, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building 130

Building Metadata MongoDB MySQL 130

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Monte Carlo

APRIL 7, 2022

Did you ever wish you had a pause button for broken data pipelines? Monte Carlo is excited to announce the release of a new suite of data observability capabilities to help data teams automatically stop broken data pipelines at the orchestration layer — before they impact the business.

Data Pipeline 52

Data Pipeline Portfolio Retail Metadata 52

Data Engineering Podcast

NOVEMBER 6, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building 130

Building IT Metadata MongoDB 130

Jesse Anderson

NOVEMBER 14, 2023

The mission of the data discovery team is twofold: 1) The data team must discover the data in the IT landscape 2) The data team must make the data in the IT landscape discoverable to the operations team Let’s unpack this. 1) The data discovery team must work on discovering the IT landscape.

Metadata 147

Metadata Data Science Big Data Data 147

Marc Lamberti

SEPTEMBER 22, 2023

Stemming from this analogy, “you” is the orchestrator in data orchestration, and the recipe is the data pipeline. It was created in 2014 by Airbnb and has since been widely adopted by the data engineering community, primarily as it was the first orchestrator allowing to author data pipelines programmatically.

Data Pipeline 130

Data Pipeline Python Metadata Database 130

Snowflake

MAY 15, 2024

We’re excited to announce today that Snowflake has invested in Metaplane , a leading end-to-end data observability platform that helps data teams improve the quality and performance of their data. Metaplane ensures that every company can trust the data that powers their business.

Cloud 110

Cloud Metadata Data Pipeline Government 110

Data Engineering Podcast

DECEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100