Data Engineering Podcast

FEBRUARY 19, 2023

Your host is Tobias Macey and today I'm interviewing Ryan Blue about the evolution and applications of the Iceberg table format and how he is making it more accessible at Tabular Interview Introduction How did you get involved in the area of data management?

IT 147

IT Data Lake Metadata Data Warehouse 147

Monte Carlo

AUGUST 30, 2023

You will also need to securely store and provide dbt with the necessary credentials to access your target database. Make sure logs are accessible for future reference. The Monte Carlo integrations with dbt Core and dbt Cloud provides streamlined notifications, troubleshooting, metadata management, and more.

SQL 52

SQL Datasets Database High Quality Data 52

Monte Carlo

JULY 11, 2023

Inconsistencies may arise due to variations in data formats, coding schemes, or definitions used by different systems or data providers. Accurate data without the necessary context or metadata can be challenging to interpret or use effectively.

Data Cleanse 40

Data Cleanse Datasets Data Governance Government 40

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 103

Government Accessible Accessibility Cloud 103

Tweag

NOVEMBER 1, 2023

But if this function calls other functions internally, the parameters to the inner call aren’t necessarily accessible. Instead of writing functions that return records and pipe them together, we write records directly where all fields might not have a definition yet. You need to bubble them up to the root function’s argument.

Metadata 114

Metadata Programming Programming Language Coding 114

Netflix Tech

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. This abstraction simplifies data access, enhances the reliability of our infrastructure, and enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort.

Bytes 99

Bytes Metadata Database Data 99

Netflix Tech

DECEMBER 3, 2022

Workflow Definitions Below you can see a typical file structure of a sample workflow package written in SparkSQL. ??? In every sample workflow package there are three workflow definition files that work together to provide flexible functionality. Attributes are set via Metacat , which is a Netflix internal metadata management platform.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Monte Carlo

JULY 10, 2023

That’s where the definition of data completeness comes in. For example, let’s say a streaming provider has 10 million overall subscribers who can access its content. Data profiling Data profiling uses a tool or programming language to surface metadata about your data. The same is true with data.

Data Collection 40

Data Collection Data Governance Government Data 40

François Nguyen

JANUARY 18, 2021

In every step,we do not just read, transform and write data, we are also doing that with the metadata. Every data governance policy about this topic must be read by a code to act in your data platform (access management, masking, etc.) Who has an access to this Data ? Last part, it was added the data security and privacy part.

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

ThoughtSpot

JUNE 28, 2023

Getting started with version control APIs To begin, ensure that the following prerequisites are met: You have admin access (can administer Thoughtspot privilege) to connect ThoughtSpot to a Git repository and deploy commits. You have a Git repository and a branch that can be used as a default branch in ThoughtSpot.

Metadata 97

Metadata Accessible Accessibility Coding 97

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 135

MySQL Metadata Data Validation Coding 135

Monte Carlo

MAY 14, 2024

Depending on the quantity of data flowing through an organization’s pipeline — or the format the data typically takes — the right modern table format can help to make workflows more efficient, increase access, extend functionality, and even offer new opportunities to activate your unstructured data.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Monte Carlo

OCTOBER 2, 2024

Something’s definitely wrong. This involves documenting many parts of the data’s journey, including where it was stored, who owned it , and how it was uploaded, modified, and accessed. Marquez is a metadata service that implements the OpenLineage API. But as you scan the numbers, something feels… off.

Python 52

Python Datasets Metadata Data 52

Pinterest Engineering

OCTOBER 2, 2024

Catalog/Metadata Management , which is responsible for storing and serving table metadata, including ownership, schemas, SLAs, etc., and also providing runtime metadata about the platform Security and Auditability , which ensures GDPR compliance, advanced levels of auditability, security, AAA, etc.

Data Management 56

Data Management Management Metadata SQL 56

phData: Data Engineering

NOVEMBER 1, 2024

It serves as a vital protective measure, ensuring proper data access while managing risks like data breaches and unauthorized use. Challenges and Considerations Balancing data access and protection is essential as GenAI tools require broad access while still adhering to governance policies. No problem!

Government 52

Government Data Governance Finance Metadata 52

François Nguyen

MARCH 22, 2021

This is really for us the definition of a self serve platform. and he/she has different actions to execute (reading, calling a vision API, transform, create metadata, store them, etc…). TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. What you have to code is this workflow

Technology 130

Technology Architecture Google Cloud Metadata 130

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 94

Metadata Python AWS Government 94

Data Engineering Podcast

MARCH 27, 2022

What is your working definition of "data governance" and how does that influence your product focus and priorities? Acryl]([link] The modern data stack needs a reimagined metadata management platform. Acryl Data’s vision is to bring clarity to your data through its next generation multi-cloud metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). The data is protected but it is also accessible by the people who need it.

Metadata 81

Metadata Government Data Lake Data 81

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. We can access the request parameters (like pathname) by running GraphQL over HTTP. on FIELD_DEFINITION For example, it is used in a query definition as follows - type CustomerConsent { status : String ! are some entities.

Metadata 82

Metadata Coding Banking Designing 82

Knowledge Hut

MARCH 7, 2024

Two, it creates a commonality of data definitions, concepts, metadata and the like. As far as Big Data is concerned, data security should be high on their priorities as most modern businesses are vulnerable to fake data generation, especially if cybercriminals have access to the database of a business.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Monte Carlo

JANUARY 23, 2024

As the machine became smarter with the next generation of data access, we flattened everything to features and columns. Almost every business I walk into, you can talk to the marketing team and the sales team and they have different definitions of things with the same name,” John says. “I Those who don’t embrace it will be left behind.

Database-centric 52

Database-centric Pipeline-centric Metadata Unstructured Data 52

Data Engineering Podcast

MAY 18, 2021

High-growth startups use Molecula’s feature store because of its unprecedented speed, cost savings, and simplified access to all enterprise data. From feature extraction to model training to production, the Molecula feature store provides continuously updated feature access, reuse, and sharing without the need to pre-process data.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

DoorDash Engineering

DECEMBER 5, 2023

Key terminology Here are definitions and links to further documentation about the tools we used. Exposes a unique hardware definition language (HDL) for configuring compute resources. All Topics, even those without any messages, include some metadata that is backed up to disk. Terraform : Infrastructure-as-code (IaC) platform.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Cloudera

MARCH 14, 2023

The data flow life cycle with Cloudera DataFlow for the Public Cloud (CDF-PC) Data flows in CDF-PC follow a bespoke life cycle that starts with either creating a new draft from scratch or by opening an existing flow definition from the Catalog.

Data Pipeline 82

Data Pipeline Designing Kafka Metadata 82

Christophe Blefari

FEBRUARY 3, 2023

ebay, Variable Hub a data access layer for risk decisioning — Looks like a feature store but for risk topics. What's often stays is that AE is between DE and DA, so the definition is often done complementarily to other positions. The article shows our Netflix art creators are using past data to create new artworks.

BI 130

BI Google Cloud Machine Learning SQL 130

Snowflake

NOVEMBER 11, 2024

Ingest data more efficiently and manage costs For data managed by Snowflake, we are introducing features that help you access data easily and cost-effectively. This reduces the overall complexity of getting streaming data ready to use: Simply create external access integration with your existing Kafka solution.

Data Architecture 79

Data Architecture Architecture Data Lake Kafka 79

Monte Carlo

MARCH 4, 2024

To do this, data observability solutions like Monte Carlo will piece together information from query logs, data aggregates and system metadata. All of this is to say that how table formats are structured, and specifically the metadata, impacts how they can be monitored and observed. Now data teams can have their lake and eat it too.

Metadata 52

Metadata Data Lake Data Engineering Data Engineer 52

Snowflake

OCTOBER 7, 2024

However, this often introduces the complexity of managing additional infrastructure for feature authoring, building and maintaining update pipelines, and establishing workflows to access consistent and fresh features. Define an Entity: Define a Feature View: feature_df is a Snowpark DataFrame object containing your feature definition.

Management 80

Management Building Datasets Government 80

Netflix Tech

JULY 22, 2024

Since Netflix’s data tables are housed in a single data warehouse, we believe a single orchestrator should handle all workflows accessing it. A workflow definition is defined in a JSON format. Maestro combines user-supplied fields with those managed by Maestro to form a flexible and powerful orchestration definition.

Machine Learning 102

Machine Learning Coding Data Pipeline Java 102

Towards Data Science

MAY 7, 2024

meetings with sales reps from all of the SaaS data integration tooling companies and are granted 14 day access to try their wares. Now you have to decide what sorts of things to test in order to figure out definitively if the tool is the right commitment for you and the team.

ETL Tools 79

ETL Tools Metadata Data Integration Data Science 79

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables (not files/blobs) are the only API abstraction for end-users.

Big Data 78

Big Data Data Management Management Metadata 78

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. Additionally, data catalogs can centrally enforce rules of access, security, and governance, all in a way that is both automatic and visible.

Metadata 52

Metadata Government Data Data Governance 52

Monte Carlo

MAY 20, 2024

It emphasizes discoverability, accessibility, and governance of data assets, promoting an integrated approach to data management across multiple platforms and stakeholders. The catalog also supports compliance and security protocols by tracking data usage and access, making it an indispensable tool for managing data at scale.

Datasets 52

Datasets Data Government Business Analyst 52

ProjectPro

FEBRUARY 8, 2023

Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. AWS Glue then creates data profiles in the catalog, a repository for all data assets' metadata, including table definitions, locations, and other features. Developers get access to developer endpoints that they can use to work with the code.

AWS 98

AWS Scala Metadata Data Lake 98

Snowflake

JUNE 4, 2024

And to provide further flexibility — including read and write interoperability from multiple engines with centralized access — Snowflake is open sourcing Polaris Catalog in the next 90 days. Parquet Direct (private preview) allows you to use Iceberg without rewriting or duplicating Parquet files — even as new Parquet files arrive.

Government 86

Government Data Ingestion Data PostgreSQL 86