Confluent

MAY 7, 2019

Although it is the simplest way to subscribe to and access events from Kafka, behind the scenes, Kafka consumers handle tricky distributed systems challenges like data consistency, failover and load balancing. There is no way that one computer node will ever be able to ingest and process all the events that get generated in real time.

Kafka 111

Kafka Accessible Accessibility Metadata 111

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. Four pillars of event streaming. Pillar 4 – Operational plane: Event logging, DLQs and automation. To read the other articles in this series, see: Journey to Event Driven – Part 1: Why Event-First Thinking Changes Everything.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Developing event-driven pipelines is going to be a lot easier - Meet Functions! Memphis Logo]([link] Developing event-driven pipelines is going to be a lot easier - Meet Functions!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Netflix Tech

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. This abstraction simplifies data access, enhances the reliability of our infrastructure, and enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort.

Bytes 99

Bytes Metadata Database Data 99

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 96

Management Kafka Metadata Media 96

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. If you missed the event or need a refresh of what was presented, watch any Snowday session on demand. Learn more about Iceberg Tables here. Learn more.

Metadata 94

Metadata Python AWS Government 94

Cloudera

APRIL 18, 2024

Here are a couple of the biggest takeaways we had from our time at the event. In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular.

Metadata 109

Metadata Government Data Governance Data 109

Data Engineering Podcast

MARCH 18, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! Data Council Logo]([link] Join us at the event for the global data community, Data Council Austin.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

François Nguyen

JANUARY 18, 2021

In every step,we do not just read, transform and write data, we are also doing that with the metadata. Every data governance policy about this topic must be read by a code to act in your data platform (access management, masking, etc.) Who has an access to this Data ? Last part, it was added the data security and privacy part.

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

Data Engineering Weekly

NOVEMBER 3, 2024

link] Event Alert: MLOps World/ Gen AI World - Austin, TX - Nov 7-8 The Gen AI Summit, consisting of a wider group of 20,000 Engineers, AI entrepreneurs, and Scientists, will host 1,000 AI teams in Austin, TX, November 7-8. Passes include app-brain-date networking, birds of a feature, post-event parties, etc. What are you waiting for?

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62

Netflix Tech

JUNE 1, 2023

New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. It also included metadata about ads, such as ad placement and impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch.

Algorithm 138

Algorithm Metadata Kafka Systems 138

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 99

Metadata Data Pipeline Scala Data Workflow 99

Precisely

JUNE 24, 2024

They are working through organizational design challenges while also establishing foundational data management capabilities like metadata management and data governance that will allow them to offer trusted data to the business in a timely and efficient manner for analytics and AI.”

Food 94

Food Data Analytics Pharmaceutical Consulting 94

Precisely

MAY 13, 2024

Integrated data catalog for metadata support As you build out your IT ecosystem, it’s important to leverage tools that have the capabilities to support forward-looking use cases. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

ThoughtSpot

JUNE 28, 2023

Getting started with version control APIs To begin, ensure that the following prerequisites are met: You have admin access (can administer Thoughtspot privilege) to connect ThoughtSpot to a Git repository and deploy commits. You have a Git repository and a branch that can be used as a default branch in ThoughtSpot.

Metadata 97

Metadata Accessible Accessibility Coding 97

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. Yeah…I can do that already!

Process 83

Process Kafka SQL Datasets 83

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 103

Government Accessible Accessibility Cloud 103

Data Engineering Podcast

AUGUST 21, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Rockset

MARCH 18, 2024

For instance, GPT-4’s cutoff date was April 2023, so it would not be aware of any events or developments happening beyond that point of time. Real-time metadata filtering Streaming data on products in a catalog is used to generate vector embeddings as well as provide additional contextual information. What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Precisely

OCTOBER 17, 2024

This year, our annual Data Integrity Summit, Trust ’24, was better than ever – and a big part of what made the event so exciting was our first-ever Data Integrity Awards ! The data catalog has made it easy for users to find, understand, and access the trusted data they need to move forward with confidence.

Data Integration 69

Data Integration Data Governance Government Metadata 69

Data Engineering Podcast

MAY 18, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. High-growth startups use Molecula’s feature store because of its unprecedented speed, cost savings, and simplified access to all enterprise data.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

François Nguyen

MARCH 22, 2021

Data & Metadata : the data of the data product in many possible storages if needed but also the metadata (data on data) Infrastructure : you will need compute & storage but with the Serverless philisophy, we want to make it totally transparent and stay focus on the first two dimensions. What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

Snowflake

APRIL 29, 2024

This strategy must incorporate business impact analyses as well as backup and recovery plans in the event of a security incident or loss of access to data. Access Control: Snowflake allows customers to define granular permissions for user roles, minimizing the risk of unauthorized access to sensitive data.

Transportation 87

Transportation Data Governance Government Consulting 87

Data Engineering Weekly

OCTOBER 13, 2024

link] Expedia: Enhancing Data Reliability With An SLO Platform Expedia Group Technology designed a new SLO platform to enhance data reliability, leveraging Kafka for event streaming, PostgreSQL for data storage, and APIs for querying.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

ProjectPro

FEBRUARY 8, 2023

Users can schedule ETL jobs, and they can also choose the events that will trigger them. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. AWS Glue then creates data profiles in the catalog, a repository for all data assets' metadata, including table definitions, locations, and other features.

AWS 98

AWS Scala Metadata Data Lake 98

Data Engineering Podcast

MARCH 27, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Acryl]([link] The modern data stack needs a reimagined metadata management platform. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Weekly

AUGUST 25, 2024

Powerful deep learning models are becoming smarter, more accessible and cost-effective. link] Max Meldrum: Introducing datafusion-uwheel, A Native DataFusion Optimizer for Time-based Analytics µWheel is an event-driven aggregate management system for ingesting, indexing, and querying stream aggregates.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Figure 1: sales group SELECT access.

Cloud 96

Cloud Kafka Metadata SQL 96

Precisely

JANUARY 25, 2024

To avoid burdening mainframe databases with constant I/O instructions and acknowledgments and prevent latency issues, best practices call for the use of event streaming platforms like Kafka, Amazon Kinesis, Rabbit MQ, or others. Real-time or near real-time replication is most common. Best Practice 2. Best Practice 3.

Metadata 69

Metadata Relational Database Data Governance Government 69

Data Engineering Podcast

FEBRUARY 27, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Metadata repository Types of metadata (catalog, lineage, access, queries, etc.) Metadata repository Types of metadata (catalog, lineage, access, queries, etc.)

Designing 100

Designing Metadata Data Lake Relational Database 100

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Java 69

Java Metadata PostgreSQL Data Warehouse 69

DoorDash Engineering

DECEMBER 5, 2023

Kafka is the pub-sub layer of the Iguazu pipeline, which provides real-time event delivery at DoorDash. Orchestrator service triggers an API call against the GitHub repo for Kafka Topics, creating a pull request, or PR, for a new topic and corresponding access control list (ACL) entry.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Snowflake

OCTOBER 2, 2023

Access to a complete view of the customer is incredibly powerful. The Snowflake Data Cloud offering enables access to all of your customer data alongside infinite compute power to model it. That allows Profiles to automatically generate an identity graph and over 30 event-based user features out of the box.

Cloud 106

Cloud Building Insurance Data Engineering 106

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. For metadata management, you can configure Snowflake to manage your Iceberg data or use an external Iceberg catalog.

Metadata 87

Metadata AWS Data Lake Datasets 87

Snowflake

NOVEMBER 11, 2024

Ingest data more efficiently and manage costs For data managed by Snowflake, we are introducing features that help you access data easily and cost-effectively. This reduces the overall complexity of getting streaming data ready to use: Simply create external access integration with your existing Kafka solution.

Data Architecture 79

Data Architecture Architecture Data Lake Kafka 79