Pinterest Engineering

JULY 15, 2024

David Chang; Staff Software Engineer | Develocity, formerly known as Gradle Enterprise, is a powerful tool that speeds up local and CI build time, helps troubleshoot your builds, and analyzes your data. At Pinterest, we have a dedicated team, Mobile Builds, and we ensure that developers can build fast and often.

Building 64

Building Metadata Software Engineer Software Engineering 64

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. Building a real-time, contextual and trustworthy knowledge base for AI applications revolves around RAG pipelines.

Cloud 64

Cloud Building Metadata Kafka 64

Tweag

JULY 26, 2023

The vast majority of the Rust projects are using Cargo as a build tool. Cargo is great when you are developing and packaging a single Rust library or application, but when it comes to a fast-growing and complex workspace, one could be attracted to the idea of using a more flexible and scalable build system.

Building 97

Building Metadata Coding Project 97

Lyft Engineering

SEPTEMBER 6, 2023

Our team, the Developer Infrastructure team, aims to build the best tools to enable microservice owners (our “customers”) to reliably and quickly test changes in a local and/or end-to-end environment. Routing overrides metadata: embed metadata in API request headers defining which offloaded deployment the request will get routed to.

Building 81

Building Metadata Electronics Engineering 81

Lyft Engineering

JUNE 28, 2023

On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. In this blog post, we will discuss what we built in support of that goal and some of the lessons we learned along the way. To meet the needs of our customers, we kicked off the Real-time Machine Learning with Streaming initiative.

Machine Learning 116

Machine Learning Building Metadata Kafka 116

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 82

Metadata BI Data Lake Business Intelligence 82

Christophe Blefari

JUNE 21, 2024

Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. But what is doing Tabular?

Metadata 147

Metadata Data Warehouse BI Scala 147

dbt Developer Hub

SEPTEMBER 14, 2021

Give Jeremy a win and check out the blog he just posted on why this matters even more leading up to ?dbt dbt build: Did you catch our teaser last month at Staging ? Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! Things to Observe ?

Metadata 52

Metadata Data Pipeline Python Cloud 52

Cloudera

AUGUST 24, 2022

CML empowers organizations to build and deploy machine learning and AI capabilities for business at scale, efficiently and securely, anywhere they want. The rest of this blog post will focus on providing instructions for a CML administrator to customize an ML runtime by adding Zeppelin as a new editor. . Prerequisites. docker.io).

Machine Learning 82

Machine Learning Building Metadata Programming Language 82

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. So which static site generator to choose?

Engineering 40

Engineering Bytes AWS Python 40

Databand.ai

AUGUST 30, 2023

Building a Winning Data Quality Strategy: Step by Step Eitan Chazbani August 30, 2023 What Is a Data Quality Strategy? This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. This starts with building a strong business case for your data quality strategy.

Building 52

Building Data Cleanse Data Governance Datasets 52

Data Engineering Podcast

JANUARY 23, 2022

He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes. You can observe your pipelines with built in metadata search and column level lineage. You can observe your pipelines with built in metadata search and column level lineage.

Building 100

Building Management Data Pipeline Metadata 100

Data Engineering Weekly

JUNE 16, 2024

One of the big benefits of having a Hive Meastore catalog is that it enables many query engines to build executing engines on top of it, which creates a strong ecosystem. Along the same line, NVIDIA open sources nmotran4 give developers a free, scalable way to generate synthetic data that can help build powerful LLMs.

Data Engineer 104

Data Engineer Data Engineering Engineering Kafka 104

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology. Conclusion.

Data Pipeline 117

Data Pipeline Building Manufacturing Data Warehouse 117

Netflix Tech

OCTOBER 19, 2020

In our previous blog post we introduced Edgar, our troubleshooting tool for streaming sessions. We could also get contextual information about the streaming session by joining relevant traces with account metadata and service logs. This insight led us to build Edgar: a distributed tracing infrastructure and user experience.

Building 94

Building Transportation Java Metadata 94

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Weekly

JULY 7, 2024

Learn More → Notion: Building and scaling Notion’s data lake Notion writes about scaling the data lake by bringing critical data ingestion operations in-house. The author highlights Paimon’s consistency model by examining the metadata model. The blog gives an overview of statistical techniques used in drift detection.

Data Engineer 98

Data Engineer Data Engineering Engineering Data Lake 98

Data Engineering Weekly

JUNE 24, 2024

I guess it is okay to build something from scratch, but I was surprised it was announced on the stage as if Databricks open-source their production version of the Unity catalog. The blog contains a summary of each talk and a link to the YouTube channel with all the talks.

Data Engineer 109

Data Engineer Data Engineering Engineering Google Cloud 109

Snowflake

JULY 25, 2024

Solving the challenges of building high-quality RAG applications From the beginning, Snowflake’s mission has been to empower customers to extract more value from their data. It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata.

Unstructured Data 113

Unstructured Data Metadata Government SQL 113

Cloudera

NOVEMBER 15, 2024

The metadata-driven approach ensures quick query planning so defenders don’t have to deal with slow processes when they need fast answers. Iceberg makes query planning more efficient by storing all of the table metadata–including partitioning and file locations–in a way that’s easy for query engines to consume.

Metadata 76

Metadata Unstructured Data Data Lake Government 76

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. A comprehensive list of benefits offered by Cosmos can be found in the linked blog. divide the input video into small chunks 2.

Process 93

Process Pipeline-centric Media Metadata 93

Data Engineering Weekly

NOVEMBER 3, 2024

impactdatasummit.com Thumbtack: What we learned building an ML infrastructure team at Thumbtack Thumbtack shares valuable insights from building its ML infrastructure team. The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. - And more!

Data Engineer 62

Data Engineer Data Engineering Engineering Pipeline-centric 62

Pinterest Engineering

JANUARY 10, 2023

Architecture High level Figure 1 — architecture of the logging pipeline Schema Client side service integration will provide the metadata, and developers just need to provide the name of the log and actual log payload. To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site.

Systems 57

Systems Building Software Engineer Software Engineering 57

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Create Iceberg materialized view For the examples in this blog, we will use three tables from the TPC-DS dataset as our base tables: store_sales, customer and date_dim. Both full and incremental rebuild of the materialized view are supported.

Metadata 89

Metadata Data Warehouse BI AWS 89

LinkedIn Engineering

MARCH 21, 2023

One of the most exciting parts of our work is that we get to play a part in helping progress a skills-first labor market through our team’s ongoing engineering work in building our Skills Graph. Engineering vs PyTorch Figure 6: Sample Seed Skills Graph KGBert helps build a more accurate and complex taxonomy in less time.

Building 85

Building Recruitment Machine Learning Deep Learning 85

Netflix Tech

MARCH 25, 2019

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can push or pull.

Building 89

Building Metadata Transportation Data Ingestion 89

Data Council

MARCH 25, 2021

How to Build a Community (Fishtown Analytics) Claire Carroll's first personal blog post on community-building is a must-read. The key takeaway is that you should start with wondering 'why' you want to build a community. Here's our March 2021 roundup of links from across the web that we selected for you: 1.

Metadata 52

Metadata Management Building IT 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. A data observability platform?

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Weekly

AUGUST 25, 2024

[link] FourSquare: Modern Data Platform: An Unbundling of a Traditional Data Warehouse When building their data platform, companies face a critical decision: adopt an all-in-one solution from vendors like Databricks, Snowflake, or AWS or compose a custom platform using tools from different providers.

Data Engineer 70

Data Engineer Data Engineering Engineering Database-centric 70

Pinterest Engineering

OCTOBER 11, 2024

In Part 2 of our blog series, we described how we were able to integrate Ray(™) into our existing ML infrastructure. In this blog post, we will discuss a second type of popular application of Ray(™) at Pinterest: offline batch inference of ML models. Results are from vLLM team’s blog.

Datasets 63

Datasets Software Engineer Software Engineering Metadata 63

Cloudera

JULY 19, 2023

Introduction Ozone is an Apache Software Foundation project to build a distributed storage platform that caters to the demanding performance needs of analytical workloads, content distribution, and object storage use cases. The hardware specifications are included at the end of this blog.

Management 69

Management Metadata Datasets Architecture 69

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. It also included metadata about ads, such as ad placement and impression-tracking events. Basic with ads was launched worldwide on November 3rd.

Algorithm 138

Algorithm Metadata Kafka Systems 138

Christophe Blefari

JULY 13, 2024

It provides abstractions and tools for the translation of lakehouse table format metadata. You can also read Timo detailed post on Mixpanel blog and why SCD are the best thing for product analytics. How to build a Semantic Layer — An great small guide that gives you the things to consider when going the Semantic Layer road.

Kafka 130

Kafka AWS Data Database 130

Netflix Tech

SEPTEMBER 18, 2024

To overcome these challenges, we developed a holistic approach that builds upon our Data Gateway Platform. Chunked data can be written by staging chunks and then committing them with appropriate metadata (e.g. Instead of immediate deletion, item metadata is updated as expired with randomly jittered TTL applied to stagger deletions.

Bytes 99

Bytes Metadata Database Data 99

LinkedIn Engineering

JULY 19, 2023

Co-Authors: Sumedh Sakdeo , Lei Sun , Sushant Raikar , Stanislav Pak , and Abhishek Nath Introduction At LinkedIn, we build and operate an open source data lakehouse deployment to power Analytics and Machine Learning workloads. While functional, our current setup for managing tables is fragmented.

Big Data 78

Big Data Data Management Management Metadata 78

Knowledge Hut

NOVEMBER 28, 2023

Create React app is the best way to quickly build a new React single-page application. Create React App will transfer these files into the build folder once the application has been constructed. Styling In React, whenever an application is built, all the CSS files are concatenated into a single bundle and placed in the build folder.

Building 40

Building Metadata Coding Project 40

Cloudera

NOVEMBER 15, 2024

The metadata-driven approach ensures quick query planning so defenders don’t have to deal with slow processes when they need fast answers. Iceberg makes query planning more efficient by storing all of the table metadata–including partitioning and file locations–in a way that’s easy for query engines to consume.

Metadata 52

Metadata Unstructured Data Data Lake Government 52