Data Engineering Podcast

FEBRUARY 19, 2023

Your host is Tobias Macey and today I'm interviewing Ryan Blue about the evolution and applications of the Iceberg table format and how he is making it more accessible at Tabular Interview Introduction How did you get involved in the area of data management? We feel your pain. It ends up being anything but that.

IT 147

IT Data Lake Metadata Data Warehouse 147

Cloudyard

MARCH 30, 2023

They are used to provide Snowflake access to unstructured datafiles and supports both of internal or external stage Query a directory table helps to retrieve the snowflake hosted file URL for each file present in stage. Directory tables metadata should be refreshed automatically when underlying stage gets updated.

Unstructured Data 52

Unstructured Data Accessibility Accessible Cloud Storage 52

Cloudera

OCTOBER 10, 2024

Users may want to perform table maintenance functions, like expiring snapshots, removing old metadata files, and deleting orphan files, to optimize storage utilization and improve performance. The open data lakehouse is quickly becoming the standard architecture for unified multifunction analytics on large volumes of data.

IT 84

IT Data Lake Metadata Data Warehouse 84

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. Results are stored in git and their database, together with benchmarking metadata. Spare Cores attempts to make it easier to compare prices across cloud providers. Source: Spare Cores. Tech stack. Benchmarking tools.

Cloud 273

Cloud Metadata AWS Cloud Computing 273

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. And don’t forget to thank them for their continued support of this show! Why do you think data observability has really taken off?

IT 100

IT Metadata MongoDB MySQL 100

Precisely

APRIL 11, 2023

Do I have access to the data? What do I need to do to access the data now and make it easy to find and access in the future? It’s critical that business analysts have the data they need and that IT has the appropriate metadata associated with those datasets for seamless replication into the cloud.

Accessibility 52

Accessibility Accessible Metadata Data Integration 52

Data Engineering Podcast

JULY 31, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Because of its centrality to your data systems it is valuable for debugging, governance, understanding context, and myriad other purposes.

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

MARCH 10, 2021

ConsoleMe: A Central Control Plane for AWS Permissions and Access By Curtis Castrapel , Patrick Sanders , and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. They are the one-stop-shop for cloud permissions and access. This happened for us at Netflix.

AWS 102

AWS Accessibility Accessible Cloud 102

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 83

Metadata BI Data Lake Business Intelligence 83

Cloudera

JUNE 7, 2022

Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system. As big data has evolved to cloud data and now to hybrid data, it has gotten more complex for businesses to access, use and create value from it. Clearly, hybrid data presents a massive opportunity and a tough challenge.

IT 112

IT Unstructured Data Data Architecture Government 112

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. And don’t forget to thank them for their continued support of this show! Email hosts@dataengineeringpodcast.com ) with your story.

Data Engineer 100

Data Engineer Data Engineering MongoDB Metadata 100

Precisely

OCTOBER 31, 2024

Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments. As data management grows increasingly complex, you need modern solutions that allow you to integrate and access your data seamlessly.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

dbt Developer Hub

OCTOBER 3, 2024

These formats are changing the way data is stored and metadata accessed. Iceberg Data Catalog - an open-source metadata management system that tracks the schema, partition, and versions of Iceberg tables. The metadata management and performance make them very meaningful and should be paid attention to. What is Iceberg? ​

Metadata 86

Metadata Data Lake Data Storage Accessible 86

Confluent

MAY 7, 2019

Although it is the simplest way to subscribe to and access events from Kafka, behind the scenes, Kafka consumers handle tricky distributed systems challenges like data consistency, failover and load balancing. Every developer who uses Apache Kafka ® has used a Kafka consumer at least once. Consumers and consumer groups.

Kafka 111

Kafka Accessibility Accessible Metadata 111

Data Engineering Podcast

JUNE 16, 2024

what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? What were the requirements and selection criteria that led to the selection of that combination of technologies? Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

ThoughtSpot

NOVEMBER 5, 2024

In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. Any delays in metadata retrieval can negatively impact user experience, resulting in decreased productivity and satisfaction.

Metadata 52

Metadata PostgreSQL Java Database 52

Data Council

JANUARY 21, 2021

Storing Cold Metadata with Alki (Dropbox) Dropbox shared insights into Alki , the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). Here's our January 2021 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata Cloud AWS Database 52

Cloudera

NOVEMBER 15, 2024

By leveraging Apache Iceberg, an open table format designed for high-performance analytics on massive volumes of data, cyber defenders can access all of their data and conduct investigations with greater speed and efficiency. The problem isn’t just the volume of the data, but also how difficult it is to manage and make sense of it.

Metadata 77

Metadata Unstructured Data Data Lake Government 77

Cloudera

JANUARY 21, 2021

With CDW, as an integrated service of CDP, your line of business gets immediate resources needed for faster application launches and expedited data access, all while protecting the company’s multi-year investment in centralized data management, security, and governance. Is there a better option than the obvious ones? Must you be: .

IT 93

IT Data Lake Data Warehouse Cloud Storage 93

Edureka

JULY 16, 2024

With the CLI, users can automate routine tasks via scripts; access and manipulate files more powerfully than from a graphical interface (GUI); and configure software that may not readily have an available GUI. Productivity: Makes using Salesforce easier by providing a faster and more convenient path to access functions in Salesforce.

IT 52

IT Cloud Computing Coding Project 52

Netflix Tech

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. This abstraction simplifies data access, enhances the reliability of our infrastructure, and enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort.

Bytes 99

Bytes Metadata Database Data 99

AltexSoft

AUGUST 29, 2023

Such an object storage model allows metadata tagging and incorporating unique identifiers, streamlining data retrieval and enhancing performance. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Weekly

JUNE 16, 2024

link] Yousry Mohamed: Delta Lake Liquid Clustering — A visual explanation Liquid clustering liberates the hive-style static partitioning and organizes the data layout from the accessing pattern. Apply for early access and achieve effective data orchestration in under 10 minutes!

Data Engineer 104

Data Engineer Data Engineering Engineering Kafka 104

Cloudera

SEPTEMBER 28, 2021

Cloudera Data platform ( CDP ) provides a Shared Data Experience ( SDX ) for centralized data access control and audit in the Enterprise Data Cloud. The Ranger Authorization Service (RAZ) is a new service added to help provide fine-grained access control (FGAC) for cloud storage. Changes with file access control .

Accessibility 62

Accessibility Accessible Cloud Cloud Storage 62

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses. Summary This podcast started almost exactly six years ago, and the technology landscape was much different than it is now.

Data Engineer 147

Data Engineer Data Engineering Engineering PostgreSQL 147

RandomTrees

SEPTEMBER 17, 2024

Understanding the Object Hierarchy in Metastore Identifying the Admin Roles in Unity Catalog Unveiling Data Lineage in Unity Catalog: Capture and Visualize Simplifying Data Access using Delta Sharing 1. Enhanced Data Security With its robust security model, Unity Catalog provides granular access control and compliance with industry standards.

Data Governance 52

Data Governance Government Metadata Machine Learning 52

ThoughtSpot

OCTOBER 9, 2023

How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance. Our data catalog integrations allow you to tap into this metadata wealth and surface it in the context where it’s needed most—when conducting business analytics.

Metadata 105

Metadata Data Government Data Governance 105

Precisely

JULY 18, 2024

Key Takeaways Data Fabric is a modern data architecture that facilitates seamless data access, sharing, and management across an organization. Data management recommendations and data products emerge dynamically from the fabric through automation, activation, and AI/ML analysis of metadata. What is Data Fabric ?

Data Management 52

Data Management Management Metadata Database-centric 52

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 101

Government Accessible Accessibility Cloud 101

phData: Data Engineering

NOVEMBER 8, 2024

Then, we add another column called HASHKEY , add more data, and locate the S3 file containing metadata for the iceberg table. Hence, the metadata files record schema and partition changes, enabling systems to process data with the correct schema and partition structure for each relevant historical dataset. Why should we use it?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Podcast

NOVEMBER 13, 2022

Shane Gibson co-founded AgileData to make analytics accessible to companies of all sizes. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Building 130

Building Metadata MongoDB MySQL 130

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Do I trust my data to be ready for use in AI?

Metadata 110

Metadata Government Data Governance Data 110

Striim

AUGUST 7, 2024

Leveraging both platforms’ strengths, companies can achieve higher operational intelligence and security, making real-time anomaly detection more effective and accessible than before. This integration enhances the detection of irregular patterns and potential threats, ensuring rapid response and mitigation.

IT 52

IT Entertainment MySQL Raw Data 52

ProjectPro

JANUARY 31, 2022

As the demand for big data grows, an increasing number of businesses are turning to cloud data warehouses. The cloud is the only platform to handle today's colossal data volumes because of its flexibility and scalability. Launched in 2014, Snowflake is one of the most popular cloud data solutions on the market. What Does Snowflake Do?

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

Snowflake

NOVEMBER 29, 2023

In medicine, lower sequencing costs and improved clinical access to NGS technology has been shown to increase diagnostic yield for a range of diseases, from relatively well-understood Mendelian disorders, including muscular dystrophy and epilepsy , to rare diseases such as Alagille syndrome.

Metadata 87

Metadata Healthcare Medical Data Storage 87

Snowflake

NOVEMBER 12, 2024

Managing application state and metadata Use Hybrid Tables as the system of record for application configuration, user profiles, workflow state and other metadata that needs to be accessed with high concurrency. Customers such as Siemens and PowerSchool are leveraging Hybrid Tables to track state for a wide variety of use cases.

Food 71

Food Metadata Education AWS 71

Snowflake

NOVEMBER 2, 2023

Snowflake Horizon is Snowflake’s built-in governance solution with a unified set of compliance, security, privacy, interoperability, and access capabilities. Snowflake continues to advance Snowflake Horizon with additional capabilities for compliance, security, privacy, interoperability, and access.

Metadata 106

Metadata Government AWS Medical 106