Pinterest Engineering

MARCH 26, 2025

Personalization Stack Building a Gift-Optimized Recommendation System The success of Holiday Finds hinges on our ability to surface the right gift ideas at the right time. Unified Logging System: We implemented comprehensive engagement tracking that helps us understand how users interact with gift content differently from standardPins.

Building 59

Building Engineering Algorithm Systems 59

Knowledge Hut

NOVEMBER 20, 2023

The DevOps lifecycle phases are in order from left to right, with each phase building upon the last. It is about automating the process of building, testing, deploying, and maintaining applications to reduce time-to-market for new features and functionality. Code - During this point, the code is being developed.

Utilities 97

Utilities Programming Coding Designing 97

dbt Developer Hub

APRIL 20, 2025

We expect that over the coming years, structured data is going to become heavily integrated into AI workflows and that dbt will play a key role in building and provisioning this data. We are committed to building the data control plane that enables AI to reliably access structured data from across your entire data lineage.

Structured Data 98

Structured Data SQL BI Project 98

Engineering at Meta

APRIL 28, 2025

We discovered that a flexible and incremental approach was necessary to onboard the wide variety of systems and languages used in building Metas products. Were upholding that by investing our vast engineering capabilities into building cutting-edge privacy technology. We believe that privacy drives product innovation.

Metadata 79

Metadata Data Utilities Data Warehouse 79

Towards Data Science

JANUARY 27, 2025

How to Build a Data Dashboard Prototype with Generative AI A book reading data visualization withVizro-AI This article is a tutorial that shows how to build a data dashboard to visualize book reading data taken from goodreads.com. Its still not complete and can definitely be extended and improved upon.

Building 53

Building Datasets Coding Data 53

Towards Data Science

JANUARY 30, 2025

Building more efficient AI TLDR : Data-centric AI can create more efficient and accurate models. Full code and results available here onGitHub. Moving experiment configs to a YAML, automatically saving results to a file, and having o1 write my visualization code made life mucheasier. Image byauthor. Image byauthor.

Database-centric 106

Database-centric Datasets Data Architecture 106

dbt Developer Hub

APRIL 19, 2023

This tutorial aims to solve this by providing the definitive guide to dimensional modeling with dbt. Don’t repeat yourself** : Dimensions can be easily re-used with other fact tables to avoid duplication of effort and code logic. Performing joins between fact and dimension tables are made simple through the use of surrogate keys.

Building 145

Building PostgreSQL BI Database 145

Knowledge Hut

FEBRUARY 1, 2024

Today, we’ll talk about how Machine Learning (ML) can be used to build a movie recommendation system - from researching data sets & understanding user preferences all the way through training models & deploying them in applications. How to Build a Movie Recommendation System in Python?

Systems 98

Systems Entertainment Algorithm Datasets 98

The Pragmatic Engineer

AUGUST 10, 2023

Ayhan visualized this data and observed a definite fall in all metrics: page views, visits, questions asked, votes. Q&A activity is definitely down: the company is aware of this metric taking a dive, and said they’re actively working to address it. When it comes to GenAI, Stack Overflow for Teams is getting a lot more love.

Retail 207

Retail Utilities Software Engineering Software Engineer 207

Towards Data Science

JANUARY 30, 2025

That said, this tutorial aims to introduce airflow-parse-bench , an open-source tool I developed to help data engineers monitor and optimize their Airflow environments, providing insights to reduce code complexity and parsetime. Parsing occurs every time Airflow processes your Python files to build the DAGs dynamically.

Python 48

Python Coding Google Cloud Database 48

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130

The Pragmatic Engineer

FEBRUARY 23, 2023

Chris Lee is the founder of US-based Launch School , which is one of the “anti bootcamp coding schools,” and an organization which impresses me. As a coding school operator, Chris has a unique perspective that gives him insight into lots of different companies and engineering departments.

Software Engineering 276

Software Engineering Software Engineer Recruitment Portfolio 276

Christophe Blefari

MARCH 1, 2023

a macro — a macro is a Jinja function that either do something or return SQL or partial SQL code. In a nutshell the dbt journey starts with sources definition on which you will define models that will transform these sources to something else you'll need in your downstream usage of the data.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Knowledge Hut

DECEMBER 22, 2023

For this feature, Python encloses certain code editors and python IDEs used for software development say, Python itself. This article looks at the top python IDEs and code editors along with their features, pros, and cons and discusses the best suited for writing Python codes. What is a Code Editor?

Python 97

Python Coding Programming Language Data Science 97

Netflix Tech

JANUARY 6, 2025

Part 2: Navigating Ambiguity By: VarunKhaitan With special thanks to my stunning colleagues: Mallika Rao , Esmir Mesic , HugoMarques Building on the foundation laid in Part 1 , where we explored the what behind the challenges of title launch observability at Netflix, this post shifts focus to the how.

Metadata 88

Metadata Algorithm Systems Building 88

Knowledge Hut

JANUARY 2, 2024

You will learn how to build up Kube-state-metrics system, pull and collect metrics, deploy a Prometheus server and metrics exporters, configure alerts with Alertmanager, and create Grafana dashboards. Here are a few methods you can use to safeguard inner loops: Reduce the number of metrics your code uses.

Architecture 98

Architecture Metadata Utilities Data Collection 98

Engineering at Meta

MAY 5, 2025

with PEP-484 , allow developers to specify variable types, enhancing code understanding without affecting runtime behavior. Type-checkers validate these annotations, helping prevent bugs and improving IDE functions like autocomplete and jump-to-definition. We improved pandas-stubs package type annotation coverage from 36% to over 50%.

Python 65

Python Coding Programming Project 65

Towards Data Science

AUGUST 8, 2023

Pioneering Data Observability: Data, Code, Infrastructure, & AI The four dimensions of data observability: data, code, infrastructure, and ai? Unreliable data doesn’t live in a silo… it’s impacted by all three ingredients of the data ecosystem: data + code + infrastructure. You look at the code.

Coding 98

Coding Data Software Engineering Software Engineer 98

Data Engineering Podcast

MAY 13, 2021

If you want to build a warehouse that gives you both control and flexibility then you might consider building on top of the venerable PostgreSQL project. In this episode Thomas Richter and Joshua Drake share their advice on how to build a production ready data warehouse with Postgres.

PostgreSQL 100

PostgreSQL Data Warehouse Building MySQL 100

Data Engineering Podcast

OCTOBER 2, 2021

If you need to deal with massive data, at high velocities, in milliseconds, then Aerospike is definitely worth learning about. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Building 100

Building BI Data Architecture Architecture 100

DataKitchen

FEBRUARY 20, 2025

It assesses your data, deploys production testing, monitors progress, and helps you build a constituency within your company for lasting change. Enhanced Testing & Profiling Copy & Move Tests with Ease The Test Definitions page now supports seamless test migration between test suites.

Datasets 74

Datasets Metadata Data Government 74

Towards Data Science

JUNE 2, 2023

How to improve the code quality of your dbt models with unit tests and TDD All you need to know to start unit testing your dbt SQL models Photo by Christin Hume on Unsplash If you are a data or analytics engineer, you are probably comfortable writing SQL models and testing for data quality with dbt tests. Kent Beck ?

Coding 97

Coding SQL Software Engineering Software Engineer 97

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Gone are the months of complex data wrangling and the constraints of no-code SaaS tools.

Cloud 116

Cloud Building Insurance Data Engineering 116

Data Engineering Podcast

SEPTEMBER 3, 2021

In this episode Lior Gavish, Lior Solomon, and Atul Gupte share their view of what it means to have a data platform, discuss their experiences building them at various companies, and provide advice on how to treat them like a software product. Who are the stakeholders in a data platform? When is a data platform the wrong choice?

Designing 100

Designing Building SQL BI 100

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Your first 30 days are free!

Data Lake 147

Data Lake High Quality Data Architecture Machine Learning 147

Towards Data Science

OCTOBER 9, 2024

Get started with Airbyte and Cloud Storage Coding the connectors yourself? Not only do you have to make it scalable and useful, but every architectural decision builds up over time. And building them yourself from scratch gives you full control of how you want them to behave. Building the data source.

Coding 75

Coding Cloud Storage Data Data Ingestion 75

Towards Data Science

MARCH 9, 2023

Code implementations for ML pipelines: from raw data to predictions Photo by Rodion Kutsaiev on Unsplash Real-life machine learning involves a series of tasks to prepare the data before the magic predictions take place. And that’s it.

Machine Learning 98

Machine Learning Building Datasets Big Data 98

Monte Carlo

NOVEMBER 26, 2024

Prediction: AI copilots that can complete a sentence, correct code errors, etc. And if Twitter has taught us anything, Sam Altman definitely has a lot to say.) We’re seeing teams build out vector databases or embedding models at scale. According to Tomasz, the current state of AI can be summed up in three categories.

Data Engineering 105

Data Engineering Data Engineer Engineering Unstructured Data 105

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Your first 30 days are free!

Programming 162

Programming Data Lake High Quality Data Machine Learning 162

Snowflake

OCTOBER 7, 2024

When scaling data science and ML workloads, organizations frequently encounter challenges in building large, robust production ML pipelines. Define an Entity: Define a Feature View: feature_df is a Snowpark DataFrame object containing your feature definition.

Management 98

Management Building Datasets Government 98

Netflix Tech

DECEMBER 3, 2022

One of the main reasons this feature exists is just like with food samples, to give you “a taste” of the production quality ETL code that you could encounter inside the Netflix data ecosystem. This is one way to build trust with our internal user base. " , country_code STRING COMMENT "Country code of the playback session."

Data Pipeline 144

Data Pipeline Scala Metadata Food 144

Snowflake

NOVEMBER 20, 2023

While such apps are being created at a very fast pace, there are two main challenges: Many modern powerful apps utilize containers to package and use code; however, this typically requires data to be moved from protected environments, increasing data privacy and security risk.

Utilities 122

Utilities Machine Learning Coding AWS 122

Towards Data Science

JULY 13, 2023

Building a maintainable and modular LLM application stack with Hamilton in 13 minutes LLM Applications are dataflows, use a tool specifically designed to express them LLM stacks. Hamilton is great for describing any type of dataflow , which is exactly what you’re doing when building an LLM powered application. Image from pixabay.

Building 97

Building Database-centric Database Coding 97

Snowflake

NOVEMBER 11, 2024

What if you could streamline your efforts while still building an architecture that best fits your business and technology needs? At BUILD 2024, we announced several enhancements and innovations designed to help you build and manage your data architecture on your terms. Here’s a closer look.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Data Engineering Podcast

DECEMBER 21, 2021

Summary One of the perennial challenges of data analytics is having a consistent set of definitions, along with a flexible and performant API endpoint for querying them. a framework for building analytics APIs to power your applications and BI dashboards Interview Introduction How did you get involved in the area of data management?

Data Analytics 130

Data Analytics BI Computer Science SQL 130

Data Engineering Weekly

APRIL 20, 2025

Register now → Thoughtworks: AI on Technology Radar ThoughtWorks' technology radar inspired many enterprises to build their internal tech radars, standardizing and suggesting technology, tools, and framework adoption. Plus, get your questions answered directly by Airflow experts and contributors.

Data Engineer 67

Data Engineer Data Engineering Engineering Kafka 67

The Pragmatic Engineer

AUGUST 3, 2023

These links were especially helpful for: Promotions and marketing campaigns QR codes Content sharing links that “just work” Converting desktop users to mobile ones The shutdown Dynamic links powered Firebase Invites: an app invite service where users could send app invite links to their friends, to drive installation of the app.

Metadata 189

Metadata Engineering Building Technology 189