Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Cloudyard

DECEMBER 24, 2024

In this blog, well explore Building an ETL Pipeline with Snowpark by simulating a scenario where commerce data flows through distinct data layersRAW, SILVER, and GOLDEN.These tables form the foundation for insightful analytics and robust business intelligence. SILVER Layer : Cleansed and enriched data prepared for analytical processing.

Building 52

Building Raw Data Scala Business Intelligence 52

Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. Can you describe how Datorios is used in a day-to-day workflow for someone building streaming applications on Flink?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. Check out the agenda and register at Neo4j.com/NODES.

Building 162

Building BI SQL Machine Learning 162

DataKitchen

MARCH 20, 2025

Unlocking Data Team Success: Are You Process-Centric or Data-Centric? We’ve identified two distinct types of data teams: process-centric and data-centric. Process-centric data teams focus their energies predominantly on orchestrating and automating workflows. They work in and on these pipelines.

Pipeline-centric 64

Pipeline-centric Database-centric Process Data 64

Precisely

MARCH 3, 2025

Data and process automation used to be seen as luxury but those days are gone. Lets explore the top challenges to data and process automation adoption in more detail. Almost half of respondents (47%) reported a medium level of automation adoption, meaning they currently have a mix of automated and manual SAP processes.

Process 59

Process Government Data Finance 59

Engineering at Meta

MARCH 4, 2025

Multimodal AI models capable of processing multiple different types of inputs like speech, text, and images have been transforming user experiences in the wearables space. Shane sits down with Pascal Hartig to share how his team is building foundational models for the Ray-Ban Meta glasses.

Building 71

Building Programming Engineering Technology 71

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

databricks

AUGUST 30, 2024

Learn how companies can create repeatable and scalable workflows that enable users to quickly turn GenAI innovation from experimentation to reality.

Process 119

Process Building Data 119

Striim

MARCH 21, 2025

What is Real-Time Stream Processing? To access real-time data, organizations are turning to stream processing. To access real-time data, organizations are turning to stream processing. There are two main data processing paradigms: batch processing and stream processing.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Pinterest Engineering

MARCH 26, 2025

Personalization Stack Building a Gift-Optimized Recommendation System The success of Holiday Finds hinges on our ability to surface the right gift ideas at the right time. Unified Logging System: We implemented comprehensive engagement tracking that helps us understand how users interact with gift content differently from standardPins.

Building 59

Building Engineering Algorithm Systems 59

Cloudera

DECEMBER 17, 2024

The introduction of these faster, more powerful networks has triggered an explosion of data, which needs to be processed in real time to meet customer demands. As more data is processed, carriers increasingly need to adopt hybrid cloud architectures to balance different workload demands.

Building 62

Building Telecommunication Data Architecture Architecture 62

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

Analytics Vidhya

SEPTEMBER 12, 2024

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge.

Data Pipeline 147

Data Pipeline Building Management Data 147

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. When performing research and building prototypes of the projects, what is your process for incorporating user experience into the implementation of the product?

Data Process 147

Data Process Process Data Lake High Quality Data 147

The Pragmatic Engineer

APRIL 21, 2024

A refresher on OpenAI, and on Evan Evan: how did you join OpenAI, and end up heading the Applied engineering group – which also builds ChatGPT? I do  not  have a PhD in Machine Learning, and was excited by the idea of building APIs and engineering teams. We do it through a process called  pretraining.

Engineering 363

Engineering Software Engineering Software Engineer Programming 363

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Code and raw data repository:   Version control: GitHub Heavily using GitHub Actions for things like getting warehouse data from vendor APIs, starting cloud servers, running benchmarks, processing results, and cleaning up after tuns. Tech stack.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Snowflake

NOVEMBER 11, 2024

At Snowflake BUILD , we are introducing powerful new features designed to accelerate building and deploying generative AI applications on enterprise data, while helping you ensure trust and safety. These scalable models can handle millions of records, enabling you to efficiently build high-performing NLP data pipelines.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Towards Data Science

JANUARY 27, 2025

How to Build a Data Dashboard Prototype with Generative AI A book reading data visualization withVizro-AI This article is a tutorial that shows how to build a data dashboard to visualize book reading data taken from goodreads.com. Now you can use Vizro-AI to build some charts by iterating text to form effective prompts.

Building 53

Building Datasets Coding Data 53

Engineering at Meta

JANUARY 22, 2025

In order to build high-quality data lineage, we developed different techniques to collect data flow signals across different technology stacks: static code analysis for different languages, runtime instrumentation, and input and output data matching, etc. In this blog, we will delve into an early stage in PAI implementation: data lineage.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Snowflake

MARCH 31, 2025

For image data, running distributed PyTorch on Snowflake ML also with standard settings resulted in over 10x faster processing for a 50,000-image dataset when compared to the same managed Spark solution. Many enterprises are already using Container Runtime to cost-effectively build advanced ML use cases with easy access to GPUs.

Healthcare 96

Healthcare Medical Government Food 96

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. Before building your own data architecture from scratch though, why not steal – er, learn from – what industry leaders have already figured out?

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform.   We didn’t build our applications in neat containers, but in bulky monoliths which commingled business, database, backend, and frontend logic. Our deployments were initially manual. Apache started to log like a maniac.

Engineering 345

Engineering Bytes Cloud Computing AWS 345

Christophe Blefari

JANUARY 11, 2025

Over the past four weeks, I took a break from blogging and LinkedIn to focus on building nao. DeepSeek is a model trained by the Chinese company with the same name, they directly compete with OpenAI and all to build foundational models. Models news and tour DeepSeek-v3 — It entered the space with a bang.

Data 130

Data Data Warehouse Coding Programming Language 130

dbt Developer Hub

APRIL 20, 2025

We expect that over the coming years, structured data is going to become heavily integrated into AI workflows and that dbt will play a key role in building and provisioning this data. We are committed to building the data control plane that enables AI to reliably access structured data from across your entire data lineage.

Structured Data 98

Structured Data SQL BI Project 98

Snowflake

MARCH 26, 2025

Welcome to Snowflakes Startup Spotlight, where we learn about awesome companies building businesses on Snowflake. By integrating AI agents, ROE AIs platform simplifies data processing, enabling organizations across industries to automate manual workflows and derive actionable intelligence from data.

Unstructured Data 98

Unstructured Data SQL Data Data Workflow 98

Striim

FEBRUARY 4, 2025

This recipe shows how you can build a data pipeline to read data from Salesforce and write to BigQuery.

Building 52

Building Data Pipeline Datasets Finance 52

The Pragmatic Engineer

JUNE 1, 2023

for the simulation engine Go on the backend PostgreSQL for the data layer React and TypeScript on the frontend Prometheus and Grafana for monitoring and observability And if you were wondering how all of this was built, Juraj documented his process in an incredible, 34-part blog series. You can read this here. ” stage Improve deploys.

Education 364

Education Project PostgreSQL Software Engineering 364

Snowflake

NOVEMBER 4, 2024

Now, it’s time to BUILD. Join us for BUILD 2024, a three-day global virtual conference taking place Nov. 12-15, to hear major Snowflake product announcements firsthand and to learn how to build with our latest innovations through dozens of technical sessions and hands-on labs.

Unstructured Data 105

Unstructured Data Python Machine Learning Data Pipeline 105

Jesse Anderson

FEBRUARY 11, 2025

Discover the insights he gained from academia and industry, his perspective on the future of data processing and the story behind building a next-generation graph database. Semih explains how Kuzu addresses the challenges of large graph analytics, the benefits of embeddability, and its potential for applications in AI and beyond.

Computer Science 130

Computer Science Database Design Software Engineering Software Engineer 130

databricks

NOVEMBER 26, 2024

Large language models are revolutionizing how we interact with technology by leveraging advanced natural language processing to perform complex tasks. In recent years.

Building 116

Building Technology Process Machine Learning 116