Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Cloudera

SEPTEMBER 28, 2021

Cloudera Data platform ( CDP ) provides a Shared Data Experience ( SDX ) for centralized data access control and audit in the Enterprise Data Cloud. The Ranger Authorization Service (RAZ) is a new service added to help provide fine-grained access control (FGAC) for cloud storage. Changes with file access control .

Accessible 61

Accessible Accessibility Cloud Cloud Storage 61

Snowflake

NOVEMBER 12, 2024

Managing application state and metadata Use Hybrid Tables as the system of record for application configuration, user profiles, workflow state and other metadata that needs to be accessed with high concurrency. Customers such as Siemens and PowerSchool are leveraging Hybrid Tables to track state for a wide variety of use cases.

Food 73

Food Metadata Education AWS 73

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 103

Government Accessible Accessibility Cloud 103

Ascend.io

JULY 23, 2024

AI Implementation Framework #1- Levels of Access The first framework focuses on defining levels of access for AI and automation tools. Organizations must determine the extent of access to data and code they are willing to grant these systems. Learn how we use metadata to automate 90% of manual data pipeline maintenance.]

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Pinterest Engineering

OCTOBER 2, 2024

As a data serving platform, SDS is responsible for various aspects of the data lifecycle management, including: Online Query Serving , which offers multiple data models such as graph, table, document, etc. Catalog/Metadata Management , which is responsible for storing and serving table metadata, including ownership, schemas, SLAs, etc.,

Data Management 56

Data Management Management Metadata SQL 56

Snowflake

JANUARY 23, 2024

Snowpark ML Operations: Model management The path to production from model development starts with model management, which is the ability to track versioned model artifacts and metadata in a scalable, governed manner. The Snowpark Model Registry API provides simple catalog and retrieval operations on models.

Machine Learning 121

Machine Learning Metadata Python Telecommunication 121

Knowledge Hut

APRIL 25, 2024

As the official Documentation says “ A Pod (as in a pod of whales or pea pod) is a group of one or more containers , with shared storage and network resources, and a specification for how to run the containers.” apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Snowflake

JULY 25, 2024

Snowflake Cortex Search, a fully managed search service for documents and other unstructured data, is now in public preview. It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata.

Unstructured Data 52

Unstructured Data Metadata Government Database 52

Snowflake

NOVEMBER 2, 2023

For example, customers who need a centralized store of data in large volume and variety – including JSON, text files, documents, images, and video – have built their data lake with Snowflake. For any features above in private preview, please contact your Snowflake account manager to apply for access.

Data Lake 95

Data Lake Data Warehouse Cloud Unstructured Data 95

ThoughtSpot

JUNE 28, 2023

Getting started with version control APIs To begin, ensure that the following prerequisites are met: You have admin access (can administer Thoughtspot privilege) to connect ThoughtSpot to a Git repository and deploy commits. For more information, explore our developer documentation on the new version control REST APIs for Git.

Metadata 97

Metadata Accessible Accessibility Coding 97

Rockset

APRIL 18, 2023

In the demo, you’ll see how Rockset delivers search results in 15 milliseconds over thousands of documents. Organizations have continued to accumulate large quantities of unstructured data, ranging from text documents to multimedia content to machine and sensor data. Why use vector search?

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

ProjectPro

FEBRUARY 8, 2023

Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. AWS Glue then creates data profiles in the catalog, a repository for all data assets' metadata, including table definitions, locations, and other features. Why Use AWS Glue? being data exactly matches the classifier, and 0.0 doesn't match the classifier.

AWS 98

AWS Scala Metadata Data Lake 98

Monte Carlo

AUGUST 6, 2024

They are integral to an organization’s data strategy, ensuring data accessibility, accuracy, and utility. Access Layer : Optimized for end-user querying and reporting. Undocumented Transformations : Lack of documentation can obscure the processes, making troubleshooting challenging.

Data Warehouse 52

Data Warehouse Raw Data BI Machine Learning 52

dbt Developer Hub

DECEMBER 19, 2023

The documentation will help you familiarize yourself with how our users interact with dbt. Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. New to dbt Cloud? ​

Metadata 52

Metadata Cloud Accessible Accessibility 52

Towards Data Science

FEBRUARY 6, 2024

Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Accessibility : I could easily request access to these data products. Metadata update Data products need to be understandable. Does it sound familiar?

Systems 77

Systems Raw Data Metadata Data Cleanse 77

François Nguyen

MARCH 22, 2021

and he/she has different actions to execute (reading, calling a vision API, transform, create metadata, store them, etc…). The core idea of Data Mesh is how you can develop the data usages and remove the centralized and monolitich data warehouse where you have very less access. What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

Rock the JVM

OCTOBER 1, 2023

In the first line of the file, we define syntax as proto3 which is the protobuf language version, next we specify package as com.rockthejvm.protos , this will be the namespace through which we access the case classes generated by Fs2Grpc. The Order service will be translated to a Scala trait through which we can access our function.

Scala 64

Scala Metadata Transportation Java 64

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 135

MySQL Metadata Data Validation Coding 135

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. Additionally, data catalogs can centrally enforce rules of access, security, and governance, all in a way that is both automatic and visible.

Metadata 52

Metadata Government Data Data Governance 52

Snowflake

AUGUST 22, 2023

If the data has specific access, usage and security requirements, these must be built into the data product. Finally, not to be overlooked are the metadata and documentation required to ensure the product can easily be used. Metadata and data should be easy to find for both humans and computers.

Metadata 74

Metadata Data AWS Business Analyst 74

Netflix Tech

DECEMBER 3, 2022

For more information on this and other examples please visit the Dataflow documentation page." This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps. Attributes are set via Metacat , which is a Netflix internal metadata management platform.

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Marc Lamberti

OCTOBER 20, 2023

A template engine is a library that combines templates with data models to produce documents. Airflow templates allow you to inject DAG Run and Task Instance metadata in your tasks at runtime. To know if you can use templating or not, you must look at… the documentation Let’s retake the BashOperator. Templating!

SQL 130

SQL Python Coding Metadata 130

Snowflake

NOVEMBER 1, 2023

And because the Notebook is natively integrated into Snowflake’s role-based access controls (RBAC), it’s easy to securely share and collaborate on your code and results without compromising on any enterprise data.

Building 109

Building Python SQL Programming Language 109

Tweag

NOVEMBER 1, 2023

But if this function calls other functions internally, the parameters to the inner call aren’t necessarily accessible. We can extract documentation from them, get completion in the LSP, use nickel query , and so on. Combined with field metadata, records can come close to the notion of modules in other languages. s t r i n g.

Metadata 114

Metadata Programming Programming Language Coding 114

Netflix Tech

JUNE 1, 2023

The replay traffic environment generated responses containing a standard playback manifest, a JSON document containing all the necessary information for a Netflix device to start playback. It also included metadata about ads, such as ad placement and impression-tracking events. Keep an eye out for updates on this.

Algorithm 138

Algorithm Metadata Kafka Systems 138

Cloudera

AUGUST 30, 2022

But while state and local governments seek to improve policies, decision making, and the services constituents rely upon, data silos create accessibility and sharing challenges that hinder public sector agencies from transforming their data into a strategic asset and leveraging it for the common good. . Forrester ).

Data Architecture 78

Data Architecture Architecture Data Lake NoSQL 78

Monte Carlo

OCTOBER 2, 2024

This involves documenting many parts of the data’s journey, including where it was stored, who owned it , and how it was uploaded, modified, and accessed. Marquez is a metadata service that implements the OpenLineage API. An example of a data lineage interface from Monte Carlo.

Python 52

Python Datasets Metadata Data 52

Data Engineering Podcast

NOVEMBER 27, 2022

This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. images, documents, etc.)

Data Process 100

Data Process Process Metadata Business Intelligence 100

Netflix Tech

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Bytes 94

Bytes Datasets Metadata Data 94

Rockset

NOVEMBER 14, 2022

For highly-selective search queries, Rockset uses its search indexes to locate documents matching predicates and then fetches the corresponding values from the row store. Rockset uses RocksDB as its embedded storage engine, storing documents as key-value pairs (ie: document identifier, document value).

Metadata 52

Metadata Datasets Cloud Accessible 52

Snowflake

OCTOBER 7, 2024

However, this often introduces the complexity of managing additional infrastructure for feature authoring, building and maintaining update pipelines, and establishing workflows to access consistent and fresh features. Standard Snowflake RBAC is used to control access to the Feature Store and the objects within.

Management 80

Management Building Datasets Government 80

Netflix Tech

DECEMBER 14, 2018

In the previous blog posts in this series, we introduced the N etflix M edia D ata B ase ( NMDB ) and its salient “Media Document” data model. NMDB is built to be a highly scalable, multi-tenant, media metadata system that can serve a high volume of write/read throughput as well as support near real-time queries.

Media 96

Media Database Metadata Data Schemas 96

Ascend.io

FEBRUARY 28, 2024

Through the design and maintenance of efficient data pipelines , data engineers facilitate the seamless flow and accessibility of data for AI processing. By ensuring data is well-organized and readily accessible, data engineers empower AI systems to deliver targeted and effective solutions.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Snowflake

APRIL 23, 2024

The connector provides immediate access to up-to-date ServiceNow data without the need to manually integrate against API endpoints. Customers get a fully managed service with no additional access fees set by Snowflake. It’s available with no additional access fees set by Snowflake and uses Snowflake credits for compute and storage.

Data Warehouse 99

Data Warehouse Data Integration Consulting Metadata 99

Monte Carlo

MAY 18, 2023

You may even already have access to one of them. No-code Editor : It’s possible to augment the lineage captured from machine metadata with a drag and drop no-code editor. One reviewer said, “I’m in the process of setting up OpenMetadata in our org, ingesting metadata from Athena, Superset and Dagster.

Metadata 64

Metadata Data Lake Government Data Governance 64

phData: Data Engineering

OCTOBER 12, 2022

Hosted Doc Site for Documentation One of the most powerful features of dbt can be the documentation you generate. This documentation can give different users insight into where data came from, what the profile of the data is, what the SQL looked like, and the DAG to know where the data is being used.

Cloud 52

Cloud Metadata SQL Data Warehouse 52