Data Engineering Weekly

NOVEMBER 3, 2024

link] Event Alert: MLOps World/ Gen AI World - Austin, TX - Nov 7-8 The Gen AI Summit, consisting of a wider group of 20,000 Engineers, AI entrepreneurs, and Scientists, will host 1,000 AI teams in Austin, TX, November 7-8. Passes include app-brain-date networking, birds of a feature, post-event parties, etc. What are you waiting for?

Data Engineering 63

Data Engineering Data Engineer Engineering Pipeline-centric 63

Snowflake

NOVEMBER 11, 2024

Ingest data more efficiently and manage costs For data managed by Snowflake, we are introducing features that help you access data easily and cost-effectively. This reduces the overall complexity of getting streaming data ready to use: Simply create external access integration with your existing Kafka solution.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

François Nguyen

JANUARY 18, 2021

In every step,we do not just read, transform and write data, we are also doing that with the metadata. Every data governance policy about this topic must be read by a code to act in your data platform (access management, masking, etc.) Who has an access to this Data ? Last part, it was added the data security and privacy part.

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Developing event-driven pipelines is going to be a lot easier - Meet Functions! Memphis Logo]([link] Developing event-driven pipelines is going to be a lot easier - Meet Functions!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Weekly

FEBRUARY 18, 2025

Kafka is designed for streaming events, but Fluss is designed for streaming analytics. This capability, termed Union Read, allows both layers to work in tandem for highly efficient and accurate data access. It excels in event-driven architectures and data pipelines. How do you compare Fluss with Apache Kafka?

Kafka 74

Kafka Lambda Architecture SQL Architecture 74

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. Yeah…I can do that already!

Process 83

Process Kafka SQL Datasets 83

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. This is what managing data without metadata feels like. This is what managing data without metadata feels like. Effective metadata management is no longer a luxury—it’s a necessity.

Metadata 52

Metadata IT Government High Quality Data 52

Netflix Tech

JUNE 1, 2023

New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. It also included metadata about ads, such as ad placement and impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch.

Algorithm 139

Algorithm Kafka Metadata Systems 139

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. If you missed the event or need a refresh of what was presented, watch any Snowday session on demand. Learn more about Iceberg Tables here. Learn more.

Metadata 118

Metadata Python Government AWS 118

Data Engineering Weekly

MARCH 23, 2025

Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. Data Council 2025 is set for April 22-24 in Oakland, CA. Initially a self-service platform (Nuage 1.0), it transitioned to a decentralized model (Nuage 2.0)

Data Engineering 63

Data Engineering Data Engineer Engineering Data 63

Snowflake

APRIL 11, 2024

At Snowflake, we believe in making the power of data accessible to all. Not only do we have a unique vantage point into the challenges faced by data analysts, we also possess rich metadata that feeds into Snowflake’s dedicated text-to-SQL model that Copilot leverages in combination with Mistral’s technology.

SQL 140

SQL Data Analysis AWS High Quality Data 140

François Nguyen

MARCH 22, 2021

Data & Metadata : the data of the data product in many possible storages if needed but also the metadata (data on data) Infrastructure : you will need compute & storage but with the Serverless philisophy, we want to make it totally transparent and stay focus on the first two dimensions. What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

Cloudyard

MARCH 30, 2023

They are used to provide Snowflake access to unstructured datafiles and supports both of internal or external stage Query a directory table helps to retrieve the snowflake hosted file URL for each file present in stage. Directory tables metadata should be refreshed automatically when underlying stage gets updated.

Unstructured Data 52

Unstructured Data Accessible Accessibility Cloud Storage 52

Data Engineering Weekly

FEBRUARY 23, 2025

Try Astro Free → Editor’s Note: Data Council 2025, Apr 22-24, Oakland, CA Data Council has always been one of my favorite events to connect with and learn from the data engineering community. Streamline code deployment, enhance collaboration, and ensure DevOps best practices with Astro's robust CI/CD capabilities.

Data Engineering 63

Data Engineering Data Engineer Engineering Kafka 63

Data Engineering Podcast

MARCH 18, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! Data Council Logo]([link] Join us at the event for the global data community, Data Council Austin.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Cloudera

NOVEMBER 13, 2020

Unlike traditional planners that need to consider accessing a table via a variety of types of index, Impala’s planner always starts with a full table scan and then applies pruning techniques to reduce the data scanned. Metadata Caching. See the performance results below for an example of how metadata caching helps reduce latency.

Metadata 143

Metadata Coding SQL Database 143

Cloudera

APRIL 18, 2024

Here are a couple of the biggest takeaways we had from our time at the event. In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular.

Metadata 110

Metadata Government Data Governance Data 110

Netflix Tech

NOVEMBER 28, 2018

Even with detection capabilities, there is a risk that exposed credentials can provide access to sensitive data and/or the ability to cause damage in our environment. Today, we would like to share two additional layers of security: API enforcement and metadata protection.

AWS 98

AWS Metadata Amazon Web Services Cloud 98

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. Quality problems lead to first responders unable to check into disaster sites or parents unable to access ESA funds. So our solution was to start using an intentional contract: Events.

Kafka 52

Kafka ETL Tools BI Database 52

Data Engineering Podcast

MAY 18, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. High-growth startups use Molecula’s feature store because of its unprecedented speed, cost savings, and simplified access to all enterprise data.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

MARCH 27, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Acryl]([link] The modern data stack needs a reimagined metadata management platform. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Snowflake

JUNE 5, 2024

At the same time, organizations must ensure the right people have access to the right content, while also protecting sensitive and/or Personally Identifiable Information (PII) and fulfilling a growing list of regulatory requirements. Additional built-in UI’s and privacy enhancements make it even easier to understand and manage sensitive data.

Government 116

Government Accessibility Accessible Cloud 116

Netflix Tech

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Bytes 95

Bytes Datasets Metadata Data 95

Knowledge Hut

JANUARY 2, 2024

Simply put, every item in a Kubernetes Prometheus store is a metric event that comes with a timestamp. Events are recorded in real time by Prometheus. Any event that is pertinent to your program can be included in this list, including memory usage, network activity, and specific inbound requests.

Architecture 98

Architecture Metadata Utilities Data Collection 98

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Figure 1: sales group SELECT access.

Cloud 97

Cloud Kafka Metadata SQL 97

Netflix Tech

JUNE 7, 2021

By collecting, accessing and analyzing network data from a variety of sources like VPC Flow Logs , ELB Access Logs, eBPF flow logs on the instances, etc, we can provide network insight to users and central teams through multiple data visualization techniques like Lumen , Atlas , etc.

Transportation 131

Transportation AWS Cloud Kafka 131

Netflix Tech

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. This abstraction simplifies data access, enhances the reliability of our infrastructure, and enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort.

Bytes 104

Bytes Metadata Database Data 104

Snowflake

APRIL 29, 2024

This strategy must incorporate business impact analyses as well as backup and recovery plans in the event of a security incident or loss of access to data. Access Control: Snowflake allows customers to define granular permissions for user roles, minimizing the risk of unauthorized access to sensitive data.

Transportation 105

Transportation Data Governance Government Consulting 105

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 101

Metadata Data Pipeline Scala Data Process 101

ThoughtSpot

JUNE 28, 2023

Getting started with version control APIs To begin, ensure that the following prerequisites are met: You have admin access (can administer Thoughtspot privilege) to connect ThoughtSpot to a Git repository and deploy commits. You have a Git repository and a branch that can be used as a default branch in ThoughtSpot.

Metadata 97

Metadata Accessibility Accessible Coding 97

Netflix Tech

DECEMBER 14, 2018

NMDB is built to be a highly scalable, multi-tenant, media metadata system that can serve a high volume of write/read throughput as well as support near real-time queries. under varying load conditions as well as a wide variety of access patterns; (b) scalability?—?persisting

Media 97

Media Database Metadata Data Schemas 97

Data Engineering Podcast

AUGUST 21, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Picnic Engineering

MARCH 28, 2025

We empowered analysts by giving them access and training to the same tools as engineers, dramatically increasing speed and impact. At the time, a single team was responsible for everythingdata modeling, warehousing, event streaming, master data management, and machine learning. That made sense early on, but it became a bottleneck.

Data Warehouse 40

Data Warehouse PostgreSQL Python Machine Learning 40

Pinterest Engineering

NOVEMBER 18, 2024

This is particularly useful in environments where multiple applications need to access and process the same data. This configuration ensures that if the host goes down due to an EC2® event or any other reason, it will be automatically reprovisioned.

Kafka 49

Kafka MySQL Database Software Engineering 49

Maxime Beauchemin

JANUARY 20, 2017

In retrospect, complex SCD modeling techniques are not intuitive and reduce accessibility. Data engineers are also the “librarians” of the data warehouse, cataloging and organizing metadata, defining the processes by which one files or extract data from the warehouse.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Confluent

APRIL 2, 2019

the event streaming platform built by the original creators of Apache Kafka. What do we mean by contextual event-driven applications? Well, infrastructure to support event-driven applications has been around for decades; mere messaging is nothing new. Accelerate the development of contextual event-driven applications.

Kafka 21

Kafka Java Cloud Metadata 21

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100