Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Precisely

MAY 5, 2025

To make this goal a reality, they seek out powerful customer communications management (CCM) solutions. However, theres often a debate on whether to build a custom in-house solution or purchase an enterprise-grade platform. What Does it Mean to Build or Buy a CCM Solution? Other teams build their own.

Building 58

Building Telecommunication Certification Data Security 58

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. How can you get the best results for your use case? Closing Announcements Thank you for listening!

Process 182

Process Building SQL BI 182

Cloudera

DECEMBER 17, 2024

Since 5G networks began rolling out commercially in 2019, telecom carriers have faced a wide range of new challenges: managing high-velocity workloads, reducing infrastructure costs, and adopting AI and automation. High-velocity workloads like network data are best managed on-premises, where operators have more control and can optimize costs.

Building 62

Building Telecommunication Data Architecture Architecture 62

Precisely

FEBRUARY 6, 2025

The traditional ways of operations management are over modernization and holistic approaches are now essential. Success in tackling modernization of IT operations management starts with assessing where your team is. The post New Year, New Approaches to Tackling IT Operations Management appeared first on Precisely.

IT 63

IT Management Datasets Systems 63

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. The services and systems need to be kept up to date, but so does the code that controls their behavior.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Pinterest Engineering

MARCH 26, 2025

Personalization Stack Building a Gift-Optimized Recommendation System The success of Holiday Finds hinges on our ability to surface the right gift ideas at the right time. Unified Logging System: We implemented comprehensive engagement tracking that helps us understand how users interact with gift content differently from standardPins.

Building 59

Building Engineering Algorithm Systems 59

Data Engineering Podcast

MAY 5, 2024

In this episode he explains his approach to building AI in a more human-like fashion and the emphasis on learning rather than statistical prediction. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Building 147

Building Data Lake High Quality Data Machine Learning 147

Edureka

JANUARY 21, 2025

Website Application Framework, or just “Web Framework,” is a group of tools and modules that lets web application developers create apps without worryingabout low-level things like thread management, protocols, and so on What is Flask? Use Cases Content management systems, e-commerce platforms, etc.

Python 52

Python Building Certification Database 52

The Pragmatic Engineer

OCTOBER 17, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. A €150K ($165K) grant, three people, and 10 months to build it. Internal comms: Chat: Slack Coordination / project management: Linear 3. To get full issues twice a week, subscribe here.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Data Engineering Weekly

DECEMBER 28, 2024

Enterprises are encouraged to experiment with AI, build numerous small-scale agents, learn from each, and expand their agent infrastructure over time. Investment in an Agent Management System (AMS) is crucial, as it offers a framework for scaling, monitoring, and refining AI agents.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

Jesse Anderson

MAY 5, 2025

Shane offers deep insights into the challenges and triumphs of digital transformation at The New York Times, discussing the shift from print to digital subscriptions, the critical role of data in that evolution, and the technical journey of building and scaling their data platform on GCP and BigQuery.

Unstructured Data 130

Unstructured Data Finance Metadata Architecture 130

The Pragmatic Engineer

MAY 11, 2023

Prometheus is part of the Cloud Native Foundation, membership of which indicates that it’s safe to build on top of Prometheus, as it’s actively maintained and will continue to be. Source: Grafana.org Clickhouse : log management. In the case of Coinbase, building in-house following a $65M bill was a clear no-brainer.

AWS 347

AWS Software Engineering Software Engineer Google Cloud 347

Precisely

JANUARY 9, 2025

In the beginning, there was a data warehouse The data warehouse (DW) was an approach to data architecture and structured data management that really hit its stride in the early 1990s. There was no easy way to consolidate and analyze this data to more effectively manage our business.

Data Integration 118

Data Integration Hadoop Data Warehouse Data Lake 118

The Pragmatic Engineer

OCTOBER 10, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. For some reason, Amazon didn’t have any senior engineers on the part of the EC2 team that was responsible for the EC2 instance management, and for the Windows business.

Software Engineering 327

Software Engineering Software Engineer Architecture Media 327

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). The npm or pnpm package managers do this additional check and so are slower, though arguably more reliable.

Programming Language 316

Programming Language Project Coding Engineering 316

Snowflake

NOVEMBER 11, 2024

At Snowflake BUILD , we are introducing powerful new features designed to accelerate building and deploying generative AI applications on enterprise data, while helping you ensure trust and safety. These scalable models can handle millions of records, enabling you to efficiently build high-performing NLP data pipelines.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Towards Data Science

JANUARY 27, 2025

How to Build a Data Dashboard Prototype with Generative AI A book reading data visualization withVizro-AI This article is a tutorial that shows how to build a data dashboard to visualize book reading data taken from goodreads.com. Now you can use Vizro-AI to build some charts by iterating text to form effective prompts.

Building 53

Building Datasets Coding Data 53

The Pragmatic Engineer

MARCH 12, 2024

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. This is how Brooks put it: “Programming managers have long recognized wide productivity variations between good programmers and poor ones. (.) A most interesting addition! The toolsmith. The tester.

Engineering 329

Engineering Programming Language Hospitality Programming 329

Snowflake

MARCH 31, 2025

Despite the best efforts of many ML teams, most models still never make it to production due to disparate tooling, which often leads to fragmented data and ML pipelines and complex infrastructure management. Many enterprises are already using Container Runtime to cost-effectively build advanced ML use cases with easy access to GPUs.

Healthcare 93

Healthcare Medical Government Food 93

Data Engineering Podcast

APRIL 23, 2023

In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. Can you describe what Meroxa is and the story behind it? How have the focus and goals of the platform and company evolved over the past 2 years?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. We started as teammates, and in a twist, I became Willem’s manager. Going from idea, to adding, to building and shipping this to all customers, is something you rarely see in bigger companies.

Software Engineering 314

Software Engineering Software Engineer Engineering Building 314

Striim

FEBRUARY 4, 2025

This recipe shows how you can build a data pipeline to read data from Salesforce and write to BigQuery. Step – 1 – Prep Work Setting up Salesforce as a source Make sure you have the permissions to be able to access the objects in the Salesforce account that you would like to read the data from.

Building 52

Building Data Pipeline Datasets Finance 52

ArcGIS

MARCH 3, 2025

Learn the secret of how the Migrate to Utility Network tool migrates any geodatabase to a utility network.

Utilities 110

Utilities Building Data Management Management 110

Christophe Blefari

JANUARY 11, 2025

Over the past four weeks, I took a break from blogging and LinkedIn to focus on building nao. DeepSeek is a model trained by the Chinese company with the same name, they directly compete with OpenAI and all to build foundational models. Models news and tour DeepSeek-v3 — It entered the space with a bang. Claude Sonnet 3.5

Data 130

Data Data Warehouse Coding Programming Language 130

dbt Developer Hub

APRIL 20, 2025

We expect that over the coming years, structured data is going to become heavily integrated into AI workflows and that dbt will play a key role in building and provisioning this data. We are committed to building the data control plane that enables AI to reliably access structured data from across your entire data lineage.

Structured Data 98

Structured Data SQL BI Project 98

The Pragmatic Engineer

NOVEMBER 21, 2023

In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers.   He then worked at the casual games company Zynga, building their in-game advertising platform. We dabbled in network engineering, database management, and system administration.

Engineering 345

Engineering Bytes Cloud Computing AWS 345

The Pragmatic Engineer

APRIL 21, 2024

A refresher on OpenAI, and on Evan Evan: how did you join OpenAI, and end up heading the Applied engineering group – which also builds ChatGPT? I do  not  have a PhD in Machine Learning, and was excited by the idea of building APIs and engineering teams. With this, it’s over to Evan. My questions are in italic.

Engineering 363

Engineering Software Engineering Software Engineer Programming 363