Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Knowledge Hut

SEPTEMBER 25, 2023

Work together with data scientists and analysts to understand the needs for data and create effective data workflows. Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Databand.ai

AUGUST 30, 2023

One key aspect of data orchestration is the automation of data pipeline tasks. By automating repetitive tasks, such as data extraction, transformation, and loading (ETL), organizations can streamline their data workflows and reduce the risk of human error.

Data Governance 52

Data Governance Data Pipeline Government Business Analyst 52

Databand.ai

AUGUST 30, 2023

Technical Challenges Choosing appropriate tools and technologies is critical for streamlining data workflows across the organization. Organizations need to automate various aspects of their data operations, including data integration, data quality, and data analytics.

Data Governance 40

Data Governance Data Cleanse Government Data Science 40

Monte Carlo

MARCH 24, 2023

Sure, terabytes or even petabytes of data are involved, but generally it’s not the size of the data but everything surrounding the data–workflows, access permissions, layers of dependencies–that pose data migration risks. When you know you can rely on your data, validating successful migrations is easier.

Data Warehouse 52

Data Warehouse AWS Cloud Database 52

dbt Developer Hub

MARCH 23, 2023

While we can surely rely on that overview to validate the final refactored model with its legacy counterpart, it can be less useful while we are in the middle of the process of rebuilding a data workflow, where we need to track down which are exactly the columns that are causing incompatibility issues and what is wrong with them.

Data Warehouse 64

Data Warehouse Generalist SQL Coding 64

phData: Data Engineering

JANUARY 3, 2022

This allows us to create new versions of our data sets, populate them with data, validate our data, and then redeploy our views on top of that data to use the new version of our data. This proactive approach to data validation allows you to minimize risks and get ahead of the issue.

IT 52

IT AWS Software Engineering Software Engineer 52