Ascend.io

JULY 28, 2023

Data pipelines are integral to business operations, regardless of whether they are meticulously built in-house or assembled using various tools. As companies become more data-driven, the scope and complexity of data pipelines inevitably expand. Ready to fortify your data management practice?

Data Pipeline 52

Data Pipeline Architecture Lambda Architecture Data Architecture 52

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights.

Data Pipeline 62

Data Pipeline Data Raw Data Database 62

Monte Carlo

APRIL 24, 2023

In an evolving data landscape, the explosion of new tooling solutions—from cloud-based transforms to data observability —has made the question of “build versus buy” increasingly important for data leaders. But it’s not just our data platform that impacts how we define the problem and its solution.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Monte Carlo

NOVEMBER 22, 2024

Your data engineering pipeline started simple: a few CSV exports, some Python scripts, and manual updates every week. You’re left wondering if there’s a breaking point where your DIY data solution won’t cut it anymore—and honestly, you might be there already. It means you’re scaling!

Data Engineer 52

Data Engineer Data Engineering Building Engineering 52

Grouparoo

OCTOBER 26, 2021

In today’s data-driven business world, organizations are looking for more efficient ways to leverage data from a variety of sources. For example, businesses often need to evaluate their performance based on large volumes of customer and sales data that might be stored in a variety of locations and formats.

Data Pipeline 52

Data Pipeline ETL Tools Data Warehouse ETL System 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure.

Data Pipeline 52

Data Pipeline BI Machine Learning Data Preparation 52

DataKitchen

NOVEMBER 5, 2024

The Race For Data Quality In A Medallion Architecture The Medallion architecture pattern is gaining traction among data teams. It is a layered approach to managing and transforming data. By systematically moving data through these layers, the Medallion architecture enhances the data structure in a data lakehouse environment.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

DareData

JULY 29, 2020

The Challenges of Medical Data In recent times, there have been several developments in applications of machine learning to the medical industry. Odds are that your local hospital, pharmacy or medical institution's definition of being data-driven is keeping files in labelled file cabinets, as opposed to one single drawer.

Data Pipeline 52

Data Pipeline Healthcare Medical Pipeline-centric 52

Ascend.io

APRIL 12, 2023

Right now, at this moment, are you prepared to act on your company’s data? So when we talk about making data usable, we’re having a conversation about data integrity. Data integrity is the overall readiness to make confident business decisions with trustworthy data, repeatedly and consistently. If not, why?

Data Pipeline 52

Data Pipeline Data Integration Datasets Data 52

Ascend.io

OCTOBER 28, 2024

What is Data Transformation? Data transformation is the process of converting raw data into a usable format to generate insights. It involves cleaning, normalizing, validating, and enriching data, ensuring that it is consistent and ready for analysis.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Towards Data Science

FEBRUARY 6, 2024

This generalisation makes their data models complex and cryptic and require domain expertise. Even harder to manage, a common setup within large organisations is to have several instances of these systems with some underlaying processes in charge of transmitting data among them, which could lead to duplications, inconsistencies, and opacity.

Systems 77

Systems Raw Data Metadata Data Cleanse 77

Hevo

FEBRUARY 15, 2023

After the hustle and bustle of extracting data from multiple sources, you have finally loaded all your data to a single source of truth like the Snowflake data warehouse. However, data modeling is still challenging and critical for transforming your raw data into any analysis-ready form to get insights.

Data Pipeline 59

Data Pipeline Building Raw Data Data Warehouse 59

Dataquest

NOVEMBER 4, 2019

If you’ve ever wanted to learn Python online with streaming data, or data that changes quickly, you may be familiar with the concept of a data pipeline. Data pipelines allow you transform data from one representation to another through a series of steps. We store the raw log data to a database.

Data Pipeline 52

Data Pipeline Python Building Raw Data 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. We expect complete and accurate data at the end of each run.

Data Engineer 107

Data Engineer Data Engineering Engineering Metadata 107

Ascend.io

SEPTEMBER 16, 2024

How to Design a Data Flow in 6 Steps: 1. Identify Data Sources 3. Define Data Requirements 4. Implement Data Flow Diagrams (DFDs) 5. Build & Optimize Data Pipelines 6. But what are data flows? They are the lifeblood of any data-driven organization, enabling analytics and informed decision-making.

Designing 52

Designing Cloud Computing Data Pipeline Raw Data 52

Snowflake

JUNE 6, 2024

You can now use Snowflake Notebooks to simplify the process of connecting to your data and to amplify your data engineering, analytics and machine learning workflows. Leverage your existing role-based access controls (RBAC) to manage access to notebooks and the underlying data assets to enable consistent and robust data governance.

SQL 97

SQL Python Machine Learning Data Workflow 97

Monte Carlo

AUGUST 6, 2024

Data warehouses are the centralized repositories that store and manage data from various sources. They are integral to an organization’s data strategy, ensuring data accessibility, accuracy, and utility. However, beneath their surface lies a host of invisible risks embedded within the data warehouse layers.

Data Warehouse 52

Data Warehouse Raw Data BI Machine Learning 52

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

ThoughtSpot

SEPTEMBER 6, 2023

Those coveted insights live at the end of a process lovingly known as the data pipeline. The pathway from ETL to actionable analytics can often feel disconnected and cumbersome, leading to frustration for data teams and long wait times for business users. Keep reading to see how it works. What is a SpotApp?

Data Workflow 105

Data Workflow Raw Data Data Lake Business Intelligence 105

DataKitchen

AUGUST 9, 2021

Below is our third post (3 of 5) on combining data mesh with DataOps to foster greater innovation while addressing the challenges of a decentralized architecture. We’ve talked about data mesh in organizational terms (see our first post, “ What is a Data Mesh? ”) and how team structure supports agility. Source: Thoughtworks.

Pharmaceutical 98

Pharmaceutical Raw Data Data Lake Data 98

Data Engineering Weekly

MARCH 31, 2024

Intuit: How Intuit data analysts write SQL 2x faster with the internal GenAI tool The productivity increase with GenAI is undeniable, and several startups are trying to solve the Text2SQL generation problem. My key highlight is that Excellent data documentation and “clean data” improve results.

Data Engineer 62

Data Engineer Data Engineering Engineering Scala 62

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Mastering Data Production (#3) Introduction Managing the production phase of data analytics is a daunting challenge. Overseeing multi-tool, multi-dataset, and multi-hop data processes ensures high-quality outputs.

Raw Data 50

Raw Data Data Ingestion Datasets Data 50

Ascend.io

FEBRUARY 6, 2023

If you’ve read this week’s release notes , you’ll have seen “Added ability to configure or override connection details at the Data Service level in BigQuery and Databricks Data Planes.” Data Services Listing with BigQuery, Snowflake, and Databricks. Data Service – is a group of Data Flows.

Cloud 52

Cloud Data Ingestion Raw Data Data Pipeline 52

Ascend.io

FEBRUARY 6, 2023

If you’ve read this week’s release notes , you’ll have seen “Added ability to configure or override connection details at the Data Service level in BigQuery and Databricks Data Planes.” Data Services Listing with BigQuery, Snowflake, and Databricks. Data Service – is a group of Data Flows.

Cloud 52

Cloud Data Ingestion Raw Data Data Pipeline 52

Meltano

OCTOBER 5, 2022

Interested in becoming a data engineer? The need for data experts in the U.S. job market is expected to grow by 22% in this decade, and according to LinkedIn’s 2020 report , a data engineer is listed as the 8th fastest growing job today. But what is data engineering exactly and what does a data engineer do?

Data Engineer 52

Data Engineer Data Engineering Engineering Raw Data 52

Databand.ai

AUGUST 30, 2023

Data testing tools: Key capabilities you should know Helen Soloveichik August 30, 2023 Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing and maintaining data quality. There are several types of data testing tools.

Data Cleanse 52

Data Cleanse Data Pipeline Datasets Data Validation 52

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. While data vault has many benefits, it is a sophisticated and complex methodology that can present challenges to data quality.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Snowflake

MAY 23, 2024

In this edition, we’ll learn why the founders of data tools company TDAA, Andrew Curran and Jon Farr, chose Snowflake as the platform to deliver their app Pancake , as well as the ways they’re effectively leveraging the Snowflake Native App model. For many data sources, the schema of the data source can change without warning.

Data Pipeline 88

Data Pipeline Raw Data Data Schemas Technology 88

Ascend.io

AUGUST 16, 2023

August 16, 2023 — Ascend.io , the leader in data pipeline automation, today released an economic analysis report conducted by Enterprise Strategy Group (ESG) of its Data Pipeline Automation Platform. As enterprise usage of data analytics grows, the field has become a significant area of IT expenditure.

Data Engineer 52

Data Engineer Data Engineering Engineering Data Pipeline 52

Cloudyard

JUNE 11, 2024

Read Time: 1 Minute, 48 Second RETRY LAST: In modern data workflows, tasks are often interdependent, forming complex task chains. Ensuring the reliability and resilience of these workflows is critical, especially when dealing with production data pipelines. Task B: Transforms the data in the staging table.

Raw Data 52

Raw Data Aggregated Data Data Pipeline Data Workflow 52

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Modern platforms like Redshift , Snowflake , and BigQuery have elevated the data warehouse model.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Data Engineering Podcast

DECEMBER 11, 2021

In order to make it easier for developers to build customer profiles in a way that respects their privacy Serge Huber helped to create the Apache Unomi framework as an open source customer data platform. Struggling with broken pipelines? Missing data? Start trusting your data with Monte Carlo today! Stale dashboards?

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

The Modern Data Company

FEBRUARY 15, 2023

You can’t blame companies for playing the cards they’re given, but now data debt is costing companies more than they think, even when solutions seem to be working…for now. What Causes Data Debt? Data debt is similar to technical debt. Data, unlike technology, seems simple on the surface.

Government 98

Government Data Governance Raw Data Data 98

Striim

NOVEMBER 17, 2023

In today’s data-driven world, the ability to leverage real-time data for machine learning applications is a game-changer. Real-time data processing in the world of machine learning allows data scientists and engineers to focus on model development and monitoring.

Machine Learning 52

Machine Learning Data Process PostgreSQL Process 52

Monte Carlo

OCTOBER 5, 2023

Data is pretty useless if you can’t act on it, but how do you share insights without crossing privacy lines or worrying that your counterparty mishandles the data? Keep reading to learn how data clean rooms work, leading providers to consider, and how not to get lost with spurious insights by embracing data observability.

Raw Data 52

Raw Data Medical Manufacturing AWS 52