Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. Data lakes are notoriously complex. Materialize]([link] You shouldn't have to throw away the database to build with fast-changing data.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Podcast

JANUARY 30, 2022

Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. You can observe your pipelines with built in metadata search and column level lineage.

Data Engineering 100

Data Engineering Data Engineer Engineering Python 100

Data Engineering Podcast

APRIL 5, 2021

It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Data Engineering Weekly

DECEMBER 16, 2024

Grab’s Metasense , Uber’s DataK9 , and Meta’s classification systems use AI to automatically categorize vast data sets, reducing manual efforts and improving accuracy. Beyond classification, organizations now use AI for automated metadata generation and data lineage tracking, creating more intelligent data infrastructures.

Data Engineering 62

Data Engineering Data Engineer Engineering Unstructured Data 62

RandomTrees

SEPTEMBER 17, 2024

The Unity Catalog is Databricks governance solution which integrates with Databricks workspaces and provides a centralized platform for managing metadata, data access, and security. Improved Data Discovery The tagging and documentation features in Unity Catalog facilitate better data discovery.

Data Governance 52

Data Governance Government Metadata Machine Learning 52

AltexSoft

JUNE 7, 2021

A HDFS Master Node, called a NameNode , keeps metadata with critical information about system files (like their names, locations, number of data blocks in the file, etc.) and keeps track of storage capacity, a volume of data being transferred, etc. Among solutions facilitation data management are. Apache Hadoop ecosystem.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Workfall

JUNE 12, 2023

It facilitates data synchronisation, replication, real-time analytics, and event-driven processing, empowering data-driven decision-making and operational efficiency. These additional columns store metadata like timestamps, user IDs, and change types, ensuring granular change tracking and auditability.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

dbt Developer Hub

OCTOBER 14, 2021

dbt Cloud v1.1.36 - v1.1.37 ​ Changelog + docs located here. The new model timing dashboard in the run detail page helps you quickly assess job composition, order, and duration to optimize your workflows and cut costs? The Model Timing tab in dbt Cloud highlights models taking particularly long to run. Want to know why? (or

Metadata 52

Metadata BI Software Engineer Software Engineering 52

Cloudera

AUGUST 21, 2020

One is data at rest, for example in a data lake, warehouse, or cloud storage and from there they can do analytics on this data and that is predominantly around what has already happened or around how to prevent something from happening in the future.

Banking 62

Banking Kafka Cloud Storage Government 62

Cloudera

FEBRUARY 11, 2019

Disruption slows as cloud and nonrelational technology take their place beside traditional approaches , the leaders extend their lead, and distributed data approaches solidify their place as a best practice for DMSA.” Cloudera believes disruption persists around multi-cloud. Why multi-cloud?

Data Management 43

Data Management Management Metadata Government 43

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Ascend.io

DECEMBER 11, 2023

Data orchestration is the process of efficiently coordinating the movement and processing of data across multiple, disparate systems and services within a company. So, why is data orchestration a big deal? Agility and Adaptability: As businesses grow and evolve, their data needs change.

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Hepta Analytics

FEBRUARY 14, 2022

Disadvantages of a data lake are: Can easily become a data swamp data has no versioning Same data with incompatible schemas is a problem without versioning Has no metadata associated It is difficult to join the data Data warehouse stores processed data, mostly structured data.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Databand.ai

AUGUST 30, 2023

DataOps tools should provide a comprehensive data cataloging solution that allows organizations to create a centralized repository of their data assets, complete with metadata, data lineage information, and data samples. Genie manages and allocates resources for big data jobs.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Edureka

JULY 3, 2024

The Basics How Azure Data Factory Works: Quick Summary Top Features of Azure Data Factory Key Components of Azure Data Factory Azure Data Factory Data Migration: Overview Azure Data Factory: Top Use Cases FAQs Conclusion What is Azure Data Factory?

Pipeline-centric 52

Pipeline-centric Data Lake Database-centric Data Pipeline 52

AltexSoft

MARCH 14, 2023

As the volume and complexity of data continue to grow, organizations seek faster, more efficient, and cost-effective ways to manage and analyze data. In recent years, cloud-based data warehouses have revolutionized data processing with their advanced massively parallel processing (MPP) capabilities and SQL support.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Knowledge Hut

SEPTEMBER 29, 2023

Why Should You Get an Azure Data Engineer Certification? Becoming an Azure data engineer allows you to seamlessly blend the roles of a data analyst and a data scientist. One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Edureka

AUGUST 19, 2024

One of the key elements of Azure Data Factory that permits data integration between various network environments is Integration Runtime. It offers the infrastructure needed to transfer data safely between cloud and on-site data storage. The three primary varieties are Azure, Azure-SSIS, and Self-hosted.

Transportation 40

Transportation Data Integration Data Storage Utilities 40

Databand.ai

DECEMBER 29, 2022

Follow Ravit on LinkedIn 5) Priya Krishnan Head of Product Management, Data and AI at IBM Priya is an innovative, customer-focused, data-driven product executive with over 16 years of experience in global product management, strategy, and GTM roles to commercialize and monetize in-demand enterprise solutions.

BI 52

BI Consulting Data Science Data Governance 52

Ascend.io

MAY 19, 2023

In the data world, this disruption manifested in the form of cloud computing with technologies such as Redshift, Snowflake, and Spark. When issues arise, there’s no need to mine and transfer diagnostic metadata or switch between tools and interfaces. Second, it enhances governance and security.

Data Pipeline 52

Data Pipeline Data Engineering Data Engineer Media 52

Monte Carlo

SEPTEMBER 20, 2022

Here’s how Prefect , Series B startup and creator of the popular data orchestration tool, harnessed the power of data observability to preserve headcount, improve data quality and reduce time to detection and resolution for data incidents. This left Dylan’s team with a gap to fill.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Monte Carlo

JUNE 14, 2023

Data pipeline architecture typically consisted of hardcoded pipelines that cleaned, normalized, and transformed the data prior to loading into a database using an ETL pattern. With cost and physical compute/storage limitations largely lifted, data engineers started to optimize data pipeline architecture for speed and agility.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Monte Carlo

AUGUST 13, 2024

Here’s how Gartner officially defines the category of data observability tools: “Data observability tools are software applications that enable organizations to understand the state and health of their data, data pipelines, data landscapes, data infrastructures, and the financial operational cost of the data across distributed environments.

Data 52

Data Data Warehouse Data Pipeline Data Architecture 52

AltexSoft

SEPTEMBER 21, 2023

Accessible via a unified API, these new features enhance search relevance and are available on Elastic Cloud. The Elastic Stacks Elasticsearch is integral within analytics stacks, collaborating seamlessly with other tools developed by Elastic to manage the entire data workflow — from ingestion to visualization.

Engineering 52

Engineering NoSQL Programming Language Java 52

Ascend.io

MARCH 17, 2023

Kubernetes Problem As cloud applications arose, the infrastructure of virtual machines became more and more similar. Since they are decoupled from a company’s architecture, they can move data from OS to OS and from cloud to cloud. Value Catching data problems in real-time avoids costly reruns and delays.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Ascend.io

MARCH 17, 2023

Kubernetes Problem As cloud applications arose, the infrastructure of virtual machines became more and more similar. Since they are decoupled from a company’s architecture, they can move data from OS to OS and from cloud to cloud. Value Catching data problems in real-time avoids costly reruns and delays.

Data Pipeline 52

Data Pipeline Datasets Data Software Engineer 52

Ascend.io

MARCH 19, 2025

Secret Management Securely handling credentials and sensitive information is non-negotiable in modern data engineering. Pipelines frequently need access to databases, APIs, and cloud services which means handling API keys, passwords, and other secrets. Did yesterdays data load complete successfully? The solution? Automation.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

AltexSoft

NOVEMBER 7, 2022

DevOps tasks — for example, creating scheduled backups and restoring data from them. Airflow is especially useful for orchestrating Big Data workflows. Airflow is not a data processing tool by itself but rather an instrument to manage multiple components of data processing. Metadata database.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Data Engineering Weekly

DECEMBER 18, 2024

AI-embedded PCs and devices with Neural Processing Units (NPUs) enable offline AI operations and improve data privacy. Meanwhile, innovations like Google’s Edge TPU will accelerate the shift toward energy-efficient edge computing, reducing dependency on centralized cloud infrastructures.

Data Engineering 113

Data Engineering Data Engineer Engineering Data Lake 113