Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

KDnuggets

NOVEMBER 6, 2024

Build an LLM application by easily picking and dropping components and connecting them, such as a vector store, web search, memory, and custom prompt.

Coding 151

Coding Building 151

The Pragmatic Engineer

MARCH 12, 2024

” Brooks agrees with this observation, and suggests a radical solution: have as few senior programmers as possible, and build a team around each one – a bit like how a hospital surgeon leads a whole team. CI/CD : running automated tests on all changes, and deploying code to production automatically. The toolsmith.

Engineering 329

Engineering Programming Language Hospitality Programming 329

KDnuggets

MARCH 17, 2025

Do you want to create custom extensions for VS Code? Follow this guide to build, test, and publish your own extension in no time!

Coding 95

Coding Building Python 95

Snowflake

JUNE 5, 2024

Enterprises can utilize gen AI to extract more value from their data and build conversational interfaces for customer and employee applications. It provides access to industry-leading large language models (LLMs), enabling users to easily build and deploy AI-powered applications. Adopting AI solutions can enhance overall efficiency.

Coding 115

Coding Building Management Government 115

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” years ago, and it became the leading AI coding assistant almost overnight.

Software Engineering 327

Software Engineering Software Engineer Programming Language Media 327

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. fix the code # fix code 7.

Systems 141

Systems Building Java Programming Language 141

The Pragmatic Engineer

SEPTEMBER 26, 2023

Every day, there’s more code at a tech company, not less. This means more repositories are needed, which are fast enough to build and work with, but which increase fragmentation. However, monorepos result in codebases growing large, so that even checking out the code or updating to the head can be time consuming.

Cloud 316

Cloud Software Engineering Software Engineer Cloud Computing 316

Analytics Vidhya

SEPTEMBER 12, 2024

Picture this: thus, unlike many other extensions that require deep setup and constant coding, […] The post Setup Mage AI with Postgres to Build and Manage Your Data Pipeline appeared first on Analytics Vidhya. That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge.

Data Pipeline 147

Data Pipeline Building Management Data 147

The Pragmatic Engineer

JUNE 1, 2023

Instead of starting with coding, Juraj kicked off by sketching. He sketched out what he wanted the final product to look like: The sketch Juraj made, before starting any coding And he sketched how he envisioned the observability part to work: The sketch of the monitoring system Phase 1: Infrastructure (October-November).

Education 364

Education Project PostgreSQL Software Engineering 364

Snowflake

APRIL 16, 2025

Welcome to Snowflakes Startup Spotlight, where we learn about awesome companies building businesses on Snowflake. Im inspired by the idea of simplifying traditionally complex tasks like building robust data-driven applications and making them accessible to everyone. What inspires you as a founder? What inspires you as a founder?

Building 59

Building Raw Data Data Analysis Data Security 59

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Other infrastructure:  Primarily AWS ( S3 for cloud object storage, Parameter Store for hierarchical storage, Elastic Container Service ( ECS ) for container deployment and orchestration) The team manages AWS via infrastructure-as-a-code Pulumi. Tech stack.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Scott Logic

MARCH 6, 2025

While vibe coding embraces AIs ability to generate quick solutions, true progress lies in models that can acknowledge ambiguity, seek clarification, and recognise when they are out of their depth. The gist of vibe coding is simple, let your AI tools worry about the code, you just instruct (prompt) the AI to do your bidding.

Coding 104

Coding Building Utilities Programming 104

Pinterest Engineering

MARCH 26, 2025

Personalization Stack Building a Gift-Optimized Recommendation System The success of Holiday Finds hinges on our ability to surface the right gift ideas at the right time. Unified Logging System: We implemented comprehensive engagement tracking that helps us understand how users interact with gift content differently from standardPins.

Building 59

Building Engineering Algorithm Systems 59

Snowflake

SEPTEMBER 18, 2023

However, building such data apps has not been easy. Any data practitioner or product owner will attest to how it takes a lot of steps to build a data app. Bring data and ML models to life Interactive visualizations Data teams now have the ability to build a whole range of new and exciting applications that were not possible before.

Python 143

Python Building Cloud Amazon Web Services 143

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform. Backend code I wrote and pushed to prod took down Amazon.com for several hours. and hand-rolled C -code. At the time, this approach was our best effort to deliver code on the nascent web. 

Engineering 345

Engineering Bytes Cloud Computing AWS 345

The Pragmatic Engineer

OCTOBER 10, 2023

He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. From learning to code in Australia, to working in Silicon Valley How did I learn to code?

Software Engineering 327

Software Engineering Software Engineer Architecture Media 327

Engineering at Meta

DECEMBER 6, 2023

To improve people’s ability to scrutinize us, we also support the Code Verify browser extension for our web-based end-to-end encrypted messaging, to give security researchers greater confidence that the code version that they are assessing is being used globally.

Building 144

Building Designing Consulting Accessibility 144

Towards Data Science

JANUARY 27, 2025

How to Build a Data Dashboard Prototype with Generative AI A book reading data visualization withVizro-AI This article is a tutorial that shows how to build a data dashboard to visualize book reading data taken from goodreads.com. Now you can use Vizro-AI to build some charts by iterating text to form effective prompts.

Building 53

Building Datasets Coding Data 53

dbt Developer Hub

APRIL 19, 2023

Don’t repeat yourself** : Dimensions can be easily re-used with other fact tables to avoid duplication of effort and code logic. Step 1: Create model files ​ Let’s create the new dbt model files that will contain our transformation code. We can then build the OBT by running dbt run.

Building 145

Building PostgreSQL BI Database 145

Monte Carlo

MARCH 28, 2025

GitHub copilot can even code alongside you like your own pocket-sized Steve Wozniak. And its not sufficient to simply build these data + AI applications – as in any other technological discipline, you have to do it reliably, too. Code Just because you have a data problem doesnt mean you have a problem with your data.

Coding 52

Coding Systems Data Pipeline ETL Tools 52

Picnic Engineering

DECEMBER 5, 2024

Building and extending a Java plugin that integrates directly with the compiler comes with some difficulties, and additionally, we’ll discuss some challenges that come with developing and maintaining an open source plugin within the Java ecosystem. The turning point in our journey came during a routine code review. How Did We Get Here?

Java 40

Java Coding Project Building 40

Engineering at Meta

JANUARY 22, 2025

In order to build high-quality data lineage, we developed different techniques to collect data flow signals across different technology stacks: static code analysis for different languages, runtime instrumentation, and input and output data matching, etc. web endpoints, data tables, AI models) used across Meta.

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

The Pragmatic Engineer

SEPTEMBER 19, 2024

Use tech debt payments to get into the flow and stay in it A good reason to add new comments to old code before you change it is to speed up a code review. When it takes me time to learn what code does, writing something down helps me remember what I figured out. Clarifying the code is even better.

Recruitment 297

Recruitment Java Coding Project 297

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How have the recent breakthroughs in large language models (LLMs) improved your ability to build features in Zenlytic? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Weekly

DECEMBER 28, 2024

Enterprises are encouraged to experiment with AI, build numerous small-scale agents, learn from each, and expand their agent infrastructure over time. Ananth shares his journey, highlighting how AI tools have reshaped his approach to coding. The key to leveraging these agents is starting small but thinking big.

Unstructured Data 113

Unstructured Data Metadata Data Government 113

Snowflake

JANUARY 27, 2025

Experienced engineers in the database space, they wrote a lot of the code that is still powering Snowflake today. Our founders believe every engineer should write code, regardless of seniority. Transparency helps build customer trust and keeps feedback flowing. In 2012, Benoit Dageville and Thierry Cruanes founded Snowflake.

Programming Language 93

Programming Language Coding Cloud Systems 93

The Pragmatic Engineer

MAY 1, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” years ago, and it became the leading AI coding assistant almost overnight.

Software Engineering 246

Software Engineering Software Engineer Programming Language Media 246

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). Bun has other contributors, but Jared writes the lion’s share of code.

Programming Language 316

Programming Language Project Coding Engineering 316

Towards Data Science

JANUARY 30, 2025

That said, this tutorial aims to introduce airflow-parse-bench , an open-source tool I developed to help data engineers monitor and optimize their Airflow environments, providing insights to reduce code complexity and parsetime. Parsing occurs every time Airflow processes your Python files to build the DAGs dynamically.

Python 48

Python Coding Google Cloud Database 48

Edureka

JANUARY 21, 2025

Building APIs Flask is often used to make RESTful APIs that let different apps talk to each other. RESTful APIs : Build APIs to serve data for frontend apps. Conclusion Flask is an ideal framework for developers seeking simplicity, flexibility, and control in building web applications. What is Flask Used For?

Python 52

Python Building Certification Database 52

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. As a listener to the Data Engineering Podcast you can get a special discount of 20% off your ticket by using the promo code dataengpod20.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262