The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Results are stored in git and their database, together with benchmarking metadata. Benchmarking results for each instance type are stored in sc-inspector-data repo, together with the benchmarking task hash and other metadata.  There Tech stack.

Cloud 273

Cloud AWS Metadata Cloud Computing 273

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. How has your experience of building and running DataHub at LinkedIn informed your product direction for Acryl? What are your plans for Data Observability?

Metadata 100

Metadata BI Data Warehouse Government 100

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? Orchestration I mentioned modularity as a core concept of building a modern data platform in my 2021 article, but I failed to emphasize the importance of data orchestration.

Building 78

Building Transportation Data Lake Metadata 78

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. We use Snowflake as our data warehouse where we build dashboards both for internal use and for customers. So our solution was to start using an intentional contract: Events.

Kafka 52

Kafka ETL Tools BI Database 52

Knowledge Hut

JUNE 4, 2024

If you've always wanted to learn how to use event binding in Angular, you've come to the right place. This article will discuss Angular's event binding and how to apply it to our Angular project. Events are triggered after these actions are completed. How Does Event Binding Works in Angular ?

Metadata 52

Metadata Coding Project Python 52

Lyft Engineering

JUNE 28, 2023

On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases. This ensured any subtle bugs stemming from differences in runtimes would be stamped out.

Machine Learning 116

Machine Learning Building Metadata Kafka 116

Rockset

MARCH 18, 2024

For instance, GPT-4’s cutoff date was April 2023, so it would not be aware of any events or developments happening beyond that point of time. Building a real-time, contextual and trustworthy knowledge base for AI applications revolves around RAG pipelines. What are the challenges building RAG pipelines? What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 106

Cloud Building Insurance Data Engineering 106

Data Engineering Podcast

MARCH 27, 2022

In this episode Balaji Ganesan shares how his experiences building and maintaining Ranger in previous roles helped him understand the needs of organizations and engineers as they define and evolve their data governance policies and practices. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Christophe Blefari

JUNE 21, 2024

Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. But what is doing Tabular?

Metadata 147

Metadata Data Warehouse BI Scala 147

Netflix Tech

SEPTEMBER 28, 2021

The Revenue and Growth Tools (RGT) team decided to set off on a journey to build tools in an abstract manner to have solutions readily available within our organization. The traditional approach of bundling dependencies at build time lacks the flexibility we need to empower our stakeholders.

Building 97

Building Metadata Designing Coding 97

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Four pillars of event streaming. Building the KPay payment system. Putting it all together.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

JUNE 17, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What is the importance of embedding column-level lineage awareness into transformation tool vs. layering on top w/ dedicated lineage/metadata tooling?

Data Lake 130

Data Lake Metadata Machine Learning Data Architecture 130

Data Engineering Podcast

OCTOBER 28, 2019

In an effort to create a better abstraction for building data applications Nick Schrock created Dagster. In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and his vision for the future of data programming.

Building 100

Building Data Pipeline Programming Language Metadata 100

Data Engineering Podcast

JANUARY 23, 2022

He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes. You can observe your pipelines with built in metadata search and column level lineage. What have been some of the recurring themes that are discussed in those events?

Building 100

Building Management Data Pipeline Metadata 100

Netflix Tech

OCTOBER 8, 2024

Building on these foundational abstractions, we developed the TimeSeries Abstraction  — a versatile and scalable solution designed to efficiently store and query large volumes of temporal event data with low millisecond latencies, all in a cost-effective manner across various use cases. For example: {“device_type”: “ios”}.

Bytes 94

Bytes Datasets Metadata Data 94

Data Engineering Weekly

NOVEMBER 3, 2024

impactdatasummit.com Thumbtack: What we learned building an ML infrastructure team at Thumbtack Thumbtack shares valuable insights from building its ML infrastructure team. Strategic pauses and adaptability are crucial for delivering timely, relevant solutions, while transparent communication builds stakeholder trust.

Data Engineering 62

Data Engineering Data Engineer Engineering Pipeline-centric 62

Monte Carlo

MAY 31, 2022

Metadata from the data warehouse/lake and from the BI tool of record can then be used to map the dependencies between the tables and dashboards. Spark has an internal framework called QueryExecutionListeners which you can configure in Spark to listen for events where a command gets executed and then pass that command to the listener.

Data Lake 52

Data Lake Building Scala Metadata 52

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing.

Algorithm 138

Algorithm Metadata Kafka Systems 138

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 96

Management Kafka Metadata Media 96

Data Engineering Podcast

MARCH 25, 2023

The peril is that building a robust and scalable streaming architecture is always more complicated and error-prone than you think it's going to be. RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code. Don't miss out on their only event this year!

MySQL 130

MySQL Python Architecture Machine Learning 130

Data Engineering Podcast

MARCH 18, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

The Pragmatic Engineer

AUGUST 3, 2023

” Because of how useful this product is, especially for app developers building on top of it, this announcement came as a surprise. To make this migration easier and as seamless as possible, we will give developers the ability to export their deep-link metadata.” We will announce more information in Q3 2023.”

Metadata 160

Metadata Engineering Building Technology 160

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Precisely

MAY 13, 2024

If you need to build stronger, faster, and flexible data integration, use this checklist of ten key attributes to streamline your search and find the right technology for your organization, faster. #1. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Data Engineering Podcast

AUGUST 21, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Precisely

JUNE 24, 2024

They are working through organizational design challenges while also establishing foundational data management capabilities like metadata management and data governance that will allow them to offer trusted data to the business in a timely and efficient manner for analytics and AI.”

Food 94

Food Data Analytics Pharmaceutical Consulting 94

Cloudera

FEBRUARY 8, 2021

To accomplish this, ECC is leveraging the Cloudera Data Platform (CDP) to predict events and to have a top-down view of the car’s manufacturing process within its factories located across the globe. . Building a Pipeline Using Cloudera Data Engineering. Curated, quality datasets are the backbone of any advanced analytics initiative.

Data Pipeline 117

Data Pipeline Building Manufacturing Data Warehouse 117

Netflix Tech

SEPTEMBER 18, 2024

To overcome these challenges, we developed a holistic approach that builds upon our Data Gateway Platform. Chunked data can be written by staging chunks and then committing them with appropriate metadata (e.g. Instead of immediate deletion, item metadata is updated as expired with randomly jittered TTL applied to stagger deletions.

Bytes 99

Bytes Metadata Database Data 99

Snowflake

DECEMBER 12, 2023

At our recent Snowday event, we announced a wave of Snowflake product innovations for easier application development, new AI and LLM capabilities, better cost management and more. If you missed the event or need a refresh of what was presented, watch any Snowday session on demand. Learn more about Iceberg Tables here. Learn more.

Metadata 94

Metadata Python AWS Government 94

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Weekly

AUGUST 25, 2024

link] FourSquare: Modern Data Platform: An Unbundling of a Traditional Data Warehouse When building their data platform, companies face a critical decision: adopt an all-in-one solution from vendors like Databricks, Snowflake, or AWS or compose a custom platform using tools from different providers.

Data Engineering 70

Data Engineering Data Engineer Engineering Database-centric 70

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. A data observability platform?

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 2, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata Electronics MongoDB MySQL 100

Netflix Tech

SEPTEMBER 10, 2024

This call includes metadata, such as the user’s information and details about the command, such as the specific show to play. The other main use case was RENO, the Rapid Event Notification System mentioned above. With these clear benefits, we continued to build out this functionality for more devices, enabling the same efficiency wins.

Metadata 91

Metadata Coding Transportation Systems 91

François Nguyen

MARCH 22, 2021

How do we build data products ? How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum. .” What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130