Data Engineering Podcast

JANUARY 15, 2022

Summary Applications of data have grown well beyond the venerable business intelligence dashboards that organizations have relied on for decades. Given this increased level of importance it has become necessary for everyone in the business to treat data as a product in the same way that software applications have driven the early 2000s.

Engineering 100

Engineering Electronics Data Pipeline ETL Tools 100

Data Engineering Weekly

MAY 3, 2024

There is no underlying semantics for unit testing the code and data testing build-in. Lack of Dataops Support Data orchestration engines like Airflow are built on the underlying Python programming language semantics. Interactive Development The data asset building is an interactive process.

Programming Language 118

Programming Language ETL Tools Data Pipeline Coding 118

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

AltexSoft

OCTOBER 30, 2021

Explaining the difference, especially when they both work with something intangible such as data , is difficult. If you’re an executive who has a hard time understanding the underlying processes of data science and get confused with terminology, keep reading. Data science vs data engineering.

Data Engineering 98

Data Engineering Data Engineer Engineering Machine Learning 98

Knowledge Hut

DECEMBER 26, 2023

According to the World Economic Forum, the amount of data generated per day will reach 463 exabytes (1 exabyte = 10 9 gigabytes) globally by the year 2025. Thus, almost every organization has access to large volumes of rich data and needs “experts” who can generate insights from this rich data.

Data Science 97

Data Science BI Business Intelligence Machine Learning 97

Confluent

MAY 15, 2019

Red Pill Analytics was hired to first design and th en implement all the necessary data integration processes required to connect Oracle WMS Cloud with their on-premises systems. This left us with only two data integration requirements to solve: relational tables and REST APIs. Streaming data into Oracle WMS Cloud.

Kafka 92

Kafka ETL Tools Cloud Data Integration 92

Hevo

SEPTEMBER 2, 2024

As far as data pipeline construction and maintenance are concerned, ETL (Extract, Transform, Load) tools play a crucial role, and their selection determines success. When considering the market offerings, AWS Glue vs Matillion frequently stands out. In this blog, we […]

ETL Tools 40

ETL Tools AWS Data Pipeline Data 40

Ascend.io

DECEMBER 19, 2023

We’ll talk about when and why ETL becomes essential in your Snowflake journey and walk you through the process of choosing the right ETL tool. Our focus is to make your decision-making process smoother, helping you understand how to best integrate ETL into your data strategy. But first, a disclaimer.

ETL Tools 52

ETL Tools IT Data Pipeline Data Warehouse 52

Data Engineering Weekly

DECEMBER 18, 2023

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. I included this post because I often see high-pitched LinkedIn posts stating it is the human fault, especially around data quality issues.

Data Engineering 68

Data Engineering Data Engineer Engineering Food 68

Cloudera

JUNE 26, 2023

Here are some tips and tricks of the trade to prevent well-intended yet inappropriate data engineering and data science activities from cluttering or crashing the cluster. Take precaution using CDSW as an all-purpose workflow management and scheduling tool. So which open source pipeline tool is better, NiFi or Airflow?

ETL Tools 79

ETL Tools Programming Language Datasets Data Pipeline 79

Ascend.io

OCTOBER 28, 2024

What is Data Transformation? Data transformation is the process of converting raw data into a usable format to generate insights. It involves cleaning, normalizing, validating, and enriching data, ensuring that it is consistent and ready for analysis. This understanding forms the basis for effective data transformation.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Hevo

MAY 30, 2023

The healthcare industry has seen an exponential growth in the use of data management and integration tools in recent years to leverage the data at their disposal. Unlocking the potential of “Big Data” is imperative in enhancing patient care quality, streamlining operations, and allocating resources optimally.

Healthcare 52

Healthcare Data Pipeline Building Big Data 52

ProjectPro

FEBRUARY 16, 2023

ETL is a critical component of success for most data engineering teams, and with teams harnessing it with the power of AWS, the stakes are higher than ever. Data Engineers and Data Scientists require efficient methods for managing large databases, which is why centralized data warehouses are in high demand.

AWS 52

AWS Data Management ETL Tools Management 52

ProjectPro

JANUARY 24, 2023

A survey by Data Warehousing Institute TDWI found that AWS Glue and Azure Data Factory are the most popular cloud ETL tools with 69% and 67% of the survey respondents mentioning that they have been using them. What is Azure Data Factory? ADF itself does not save any data. So, let’s dive in!

AWS 52

AWS Cloud Amazon Web Services ETL Tools 52

Meltano

OCTOBER 5, 2022

Interested in becoming a data engineer? The need for data experts in the U.S. job market is expected to grow by 22% in this decade, and according to LinkedIn’s 2020 report , a data engineer is listed as the 8th fastest growing job today. But what is data engineering exactly and what does a data engineer do?

Data Engineering 52

Data Engineering Data Engineer Engineering Raw Data 52

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Modern platforms like Redshift , Snowflake , and BigQuery have elevated the data warehouse model.

Data Lake 52

Data Lake Data Warehouse ETL Tools Data Pipeline 52

Scott Logic

APRIL 22, 2024

Amazon’s goal in this space is to develop AI tools that are able to be used by non-technical users, so that organisations are not hamstrung by any bottlenecks caused by the number of data scientists they have available. In particular, with SageMaker Canvas, it’s possible to create a machine learning model entirely graphically.

BI 97

BI Software Engineering Software Engineer Algorithm 97

Databand.ai

AUGUST 30, 2023

Data testing tools: Key capabilities you should know Helen Soloveichik August 30, 2023 Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing and maintaining data quality. There are several types of data testing tools.

Data Cleanse 52

Data Cleanse Data Pipeline Datasets Data Validation 52

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Silectis

DECEMBER 15, 2020

If you’ve been following along with Silectis over the past couple of years, you are familiar with our data engineering platform, Magpie. You’re aware of the many outcomes it puts at the fingertips of data engineers, and teams of data practitioners more largely. The time to identify more insights out of huge sets of data.

Data Engineering 52

Data Engineering Data Engineer Engineering ETL Tools 52

Monte Carlo

JULY 22, 2024

Data doesn’t just flow – it floods in at breakneck speed. How do we track this tsunami of changes, ensure data integrity, and extract meaningful insights? Data versioning is the answer. In this guide, we’ll explore how data versioning enhances our ability to capitalize on data even as information evolves rapidly.

Metadata 52

Metadata Datasets ETL Tools Data 52

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. Sign up free to test out the tool today. 2 I agree; permission is a mess.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Knowledge Hut

NOVEMBER 17, 2023

Azure Data Engineers play an important role in building efficient, secure, and intelligent data solutions on Microsoft Azure's powerful platform. The position of Azure Data Engineers is becoming increasingly important as businesses attempt to use the power of data for strategic decision-making and innovation.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Knowledge Hut

MARCH 28, 2024

Data science has become one of the most trending fields today. Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 This demonstrates the increasing need for Microsoft Certified Data Engineers. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

APRIL 4, 2023

Today, I’m excited to announce Monte Carlo’s new Fivetran integration , giving mutual customers the ability to accelerate data incident detection and resolution by adding monitoring to data pipelines at the point of creation.

ETL Tools 52

ETL Tools Data Pipeline Business Intelligence Data Warehouse 52

Monte Carlo

APRIL 4, 2023

Today, I’m excited to announce Monte Carlo’s new Fivetran integration , giving mutual customers the ability to accelerate data incident detection and resolution by adding monitoring to data pipelines at the point of creation.

ETL Tools 52

ETL Tools Data Pipeline Business Intelligence Data Warehouse 52

Grouparoo

AUGUST 24, 2021

The State of Customer Data The Modern Data Stack is all about making powerful marketing and sales decisions and performing impactful business analytics from a single source of truth. Customer Data Integration makes this possible. In fact, only 34% of marketing teams feel satisfied with their customer data solutions 1.

Data Integration 52

Data Integration Data Consolidation Data Warehouse ETL Tools 52

Striim

NOVEMBER 7, 2022

We recently announced that Striim is a participant in Microsoft’s Intelligent Data Platform partner ecosystem. However, in order to make the most of Synapse and Power BI you need to reliably ingest data from disparate sources in real time. This reduces the time to value, and gets the data in the right format ahead of time.

ETL Tools 52

ETL Tools BI Java Data Pipeline 52

Monte Carlo

OCTOBER 8, 2024

In the world of data, there a lot of handoffs between various data stakeholders. But what’s the real difference between data stewards vs data analysts? The data steward makes sure everything is stored correctly, access is controlled, and the database doesn’t become a dumpster fire.

Data 52

Data Finance Data Governance Government 52

Silectis

MARCH 9, 2021

The importance of data engineering is on the rise, with organizations increasingly investing in talent and infrastructure. I caught up with a few members of the team to take note of some of the data engineering trends we anticipate seeing more of this year and beyond. – Matt Boegner , Data Architect at Silectis 2.

Data Engineering 52

Data Engineering Data Engineer Engineering ETL Tools 52

Knowledge Hut

JANUARY 30, 2024

In today's world, where data rules the roost, data extraction is the key to unlocking its hidden treasures. As someone deeply immersed in the world of data science, I know that raw data is the lifeblood of innovation, decision-making, and business progress. What is data extraction?

ETL Tools 52

ETL Tools Database-centric Data Mining Raw Data 52

Grouparoo

SEPTEMBER 7, 2021

Operational analytics is the process of creating data pipelines and datasets to support business teams such as sales, marketing, and customer support. Data analysts and data engineers are responsible for building and maintaining data infrastructure to support many different teams at companies.

ETL Tools 52

ETL Tools Data Warehouse Business Intelligence Datasets 52

AltexSoft

OCTOBER 21, 2022

Similar to Google in web browsing and Photoshop in image processing, it became a gold standard in data streaming, preferred by 70 percent of Fortune 500 companies. Apache Kafka is an open-source, distributed streaming platform for messaging, storing, processing, and integrating large data volumes in real time. What is Kafka?

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Monte Carlo

OCTOBER 25, 2024

We’re the middle children of the data revolution, born into systems promised to be ‘set it and forget it,’ taught to believe that our pipelines would run forever. The first rule of data pipelines is: they will break. The second rule of data pipelines is: THEY WILL BREAK. They won’t.

Data Pipeline 52

Data Pipeline ETL Tools Cloud Systems 52

Monte Carlo

OCTOBER 16, 2024

Monte Carlo, the data observability leader, today announced new observability features for Azure Data Factory, Informatica, and Databricks Workflows. With the new release, Monte Carlo users can gain visibility into pipeline failures, lineage and pipeline performance – all in a single pane of glass.

ETL Tools 40

ETL Tools Data Pipeline Data Coding 40

Databand.ai

JULY 5, 2023

What Are Data Observability Tools? Data observability tools are software solutions that oversee, analyze, and improve the performance of data pipelines. By employing these tools, teams can proactively detect issues before they become larger problems that affect business operations.

Data Pipeline 40

Data Pipeline Data Lake Data Warehouse Datasets 40

Monte Carlo

OCTOBER 10, 2024

One of the most underrated – but most important – roles emerging from the AI craze is AI Data Quality Analyst. As the use of AI becomes more ubiquitous across data organizations and beyond, data quality rises in importance right alongside it. Table of Contents What Does an AI Data Quality Analyst Do?

Unstructured Data 52

Unstructured Data Google Cloud Machine Learning ETL Tools 52