Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

JULY 3, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

JUNE 12, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Monte Carlo

JULY 22, 2024

While it shares similarities with software versioning, data versioning has unique characteristics specific to your data management needs. Maintaining metadata about each version. By implementing data versioning, you can create a systematic approach to managing the evolution of your data.

Metadata 52

Metadata Datasets ETL Tools Data 52

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data quality, where errors are occurring, or how a report was constructed you need to track the lineage of the data from beginning to end.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

NOVEMBER 20, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 130

Data Lake Data Ingestion MongoDB Google Cloud 130

Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Weekly

OCTOBER 6, 2024

The learning goes back to the fundamentals of pipeline design principles. Regularly review if pipelines are still required. Minimize the data used in pipelines, aka do incremental data pipeline design. Optimize pipeline schedules. Filter data effectively to make sure the query uses partition pruning.

Data Engineering 62

Data Engineering Data Engineer Engineering Data Warehouse 62

Ascend.io

DECEMBER 19, 2022

We hope the real-time demonstrations of Ascend automating data pipelines were a real treat—a long with the special edition T-Shirt designed specifically for the show (picture of our founder and CEO rocking the t-shirt below). Instead, it is a Sankey diagram driven by the same dynamic metadata that runs the Ascend control plane.

Data Ingestion 52

Data Ingestion Data Pipeline Metadata AWS 52

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Monte Carlo

AUGUST 22, 2023

Slow running data pipelines cost data teams time, money, and goodwill. They utilize excess compute, cause data quality issues, and create a poor user experience for data consumers that must wait in exasperation for data to return, dashboards to load, and AI models to update.

Data Pipeline 52

Data Pipeline Metadata Data Warehouse Utilities 52

Ascend.io

MAY 23, 2023

The data engineering world is full of tips and tricks on how to handle specific patterns that recur with every data pipeline. In general, the root of the orphaned data problem lies in the interim storage of data between the individual processing steps that make up a data pipeline.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Datakin

SEPTEMBER 13, 2021

I’m Ross from Datakin I’d like to show you a new approach to keeping your pipelines running smoothly. Datakin observes your jobs as they run, collecting metadata that helps you understand how data flows through your ecosystem. Blog Datakin in 104 seconds Written by Ross Turk on Sep 13, 2021 Hi!

Metadata 52

Metadata Datasets Data Pipeline Systems 52

Monte Carlo

AUGUST 6, 2024

Implementing a Proactive Approach: Best Practices To combat these hidden threats in your data warehouse, being proactive is essential. Implement these best practices to enhance your data integrity: Clear Documentation and Metadata Management : Ensure all your data assets and transformations are well-documented.

Data Warehouse 52

Data Warehouse Raw Data BI Machine Learning 52

Precisely

JULY 21, 2023

Whether your organization’s focus is improving the customer experience, automating operations, mitigating risk, or accelerating growth and profitability, every initiative relies on data that is trusted to be accurate, consistent, and contextualized.

Data Integration 52

Data Integration Kafka Bytes Banking 52

The Modern Data Company

JANUARY 17, 2023

These include internal data lakes, apps (1st or 3rd party), web SDK, internal or 3rd-party databases, and even spreadsheets generated within the marketing department. Without lineage or governance metadata, the quality of this data is anyone’s guess. New data is added to the catalog automatically upon ingestion.

Metadata 59

Metadata Government Data Lake Data 59

Data Engineering Weekly

JULY 2, 2023

Please send your talk proposal today, and let's collaborate to make this the most exciting, insightful, and interactive Data Eng & AI conference yet! LinkedIn write about Hoptimator for auto generated Flink pipeline with multiple stages of systems. Please watch out for this space for the conference registration page soon!!!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Towards Data Science

JULY 8, 2024

Leveraging TensorFlow Transform for scaling data pipelines for production environments Photo by Suzanne D. Williams on Unsplash Data pre-processing is one of the major steps in any Machine Learning pipeline. I have used Colab for this demo, as it is much easier (and faster) to configure the environment.

Data Preparation 46

Data Preparation Datasets Metadata Data Ingestion 46

Towards Data Science

JUNE 19, 2023

I use Compose daily at work and for my personal projects to build and manage my data pipelines, and today I’d like to show how you can build your ETL projects with Compose. Inspired by Docker Compose, this solution came in the form of a project called Meerschaum Compose. mrsm compose run Registers the pipes and syncs them one-by-one.

PostgreSQL 76

PostgreSQL SQL Python Project 76

Silectis

DECEMBER 15, 2020

We built Magpie with integrated security and data governance features including data lineage, detailed activity tracking, object-level access control, metadata management, scheduling and dependency management, comprehensive reporting, and DevOps support. The time to identify more insights out of huge sets of data.

Data Engineer 52

Data Engineer Data Engineering Engineering ETL Tools 52

Rockset

SEPTEMBER 26, 2023

Traditional approaches rely upon cascading batch-oriented data pipelines, meaning data takes hours or even days to flow through the enterprise. As a result, data made available is stale and of low fidelity. We aim to coherently feed these diverse inputs into a model with low latency and without a complex architecture.

Metadata 64

Metadata Kafka Cloud Database 64

Rockset

APRIL 18, 2023

In real-time ML, the freshness of the features, the serving latency, and the uptime and availability of the data pipeline and model matter. Generating Features for Real-Time Anomaly Detection Now download the anomaly detection dataset to the data / directory. 0 0 Ok, we have the imported data.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

AltexSoft

NOVEMBER 7, 2022

The platform went live in 2015 at Airbnb, the biggest home-sharing and vacation rental site, as an orchestrator for increasingly complex data pipelines. How data engineering works. Apache Airflow is an open-source Python -based workflow orchestrator that enables you to design, schedule, and monitor data pipelines.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Today’s LLMs can already process enormous amounts of unstructured data, automating much of the monotonous work of data science. It can show me how it built that chart, which dataset it used, and show me the metadata.”

Database-centric 52

Database-centric Pipeline-centric Metadata Unstructured Data 52

Rockset

AUGUST 26, 2022

When a change is detected, the data is ingested into a data lake hosted on Amazon S3. There, the data is converted, tagged with metadata, cleaned, and de-duplicated in preparation for queries. They demoed but quickly eliminated Snowflake.

IT 52

IT Food Aggregated Data Data Lake 52

WeCloudData

OCTOBER 19, 2021

It was recommended to pause the crawlers and enable glue metadata registry on the Kinesis level. We proposed a few designed changes, one of the most suitable method is to use Athena and QuickSight for data analytics and data visualization (See appendix for dashboard that WeCloudData team created).

Consulting 52

Consulting AWS Aggregated Data Data Collection 52

WeCloudData

OCTOBER 19, 2021

It was recommended to pause the crawlers and enable glue metadata registry on the Kinesis level. We proposed a few designed changes, one of the most suitable method is to use Athena and QuickSight for data analytics and data visualization (See appendix for dashboard that WeCloudData team created).

Consulting 52

Consulting AWS Aggregated Data Data Collection 52

Monte Carlo

AUGUST 7, 2023

Prevention: Finally, data observability also provides mechanisms to prevent data issues from happening in the first place, like placing circuit breaking in pipelines and creating visibility around the impact code changes would make on data, among other proactive measures of preventing bad data from entering your pipelines in the first place.

Coding 52

Coding Data Software Engineering Software Engineer 52

AltexSoft

OCTOBER 8, 2021

To make a more informed decision, enquire about demos and do your own in-depth research. Oracle Data Integrator, IBM InfoSphere, Snaplogic, Xplenty, and. The prevailing part of users claim that it is quite easy to configure and manage data flows with Oracle’s graphical tools. Data profiling and cleansing. Ease of use.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

Data Engineering Podcast

JANUARY 28, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. So all the files are stored in one register, and all the metadata is stored in another register.

Data 100

Data Project Data Management Electronics 100

Monte Carlo

JANUARY 23, 2023

In the case of data observability, however, a single pane of glass is integral to how efficient and effective your data team can be in its data reliability workflows. It’s great to know data moved from point A to point B, but your integration points will paint the full story of what happened to it along the way.

IT 52

IT BI Government Data Governance 52

The Modern Data Company

JANUARY 22, 2024

This mission culminates in the esteemed recognition of honorable mention in Gartner’s 2023 Magic Quadrant for Data Integration, showcasing commitment to excellence and industry leadership in the data-driven era. Data engineering excellence Modern offers robust solutions for building, managing, and operationalizing data pipelines.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

The Modern Data Company

JANUARY 22, 2024

This mission culminates in the esteemed recognition of honorable mention in Gartner’s 2023 Magic Quadrant for Data Integration, showcasing commitment to excellence and industry leadership in the data-driven era. Data engineering excellence Modern offers robust solutions for building, managing, and operationalizing data pipelines.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Precisely

JULY 15, 2024

The move to productize data also requires a way to package data products so they are easily and uniformly discoverable and consumed. Data products must be properly designed and organized to be reused across the organization. Data catalogs or data marketplaces can be used to enhance the value of your data mesh strategy.

Data Governance 72

Data Governance Data Integration Government Data 72

Ascend.io

APRIL 7, 2023

With the monolithic architectures most organizations have today, business users are stuck, constantly waiting for new data pipelines to be built or amended based on their requests. Data engineers aren’t huge fans of this paradigm either. Anyone can query the metadata any time anywhere to get the information they need.

Metadata 52

Metadata Datasets Data Governance Government 52

AltexSoft

MARCH 30, 2023

Source: The Data Team’s Guide to the Databricks Lakehouse Platform Integrating with Apache Spark and other analytics engines, Delta Lake supports both batch and stream data processing. Besides that, it’s fully compatible with various data ingestion and ETL tools.

Scala 64

Scala Data Lake BI Google Cloud 64

Monte Carlo

APRIL 20, 2023

I can surface ownership metadata and alert the relevant owners to make sure the appropriate changes are made so these breakages never happen. As the first two subsections highlight, data and people are constantly shifting. Data Governance and Compliance The only things propagating faster than data are data regulations.

Data Warehouse 52

Data Warehouse BI Data Government 52