Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. How can you get the best results for your use case?

Building 162

Building BI SQL Machine Learning 162

The Pragmatic Engineer

MARCH 12, 2024

The title of the book takes aim at the “myth” that software development can be measured in “man months,” which Brooks disproves in the pages that follow: “Cost [of the software project] does indeed vary as the product of the number of men and the number of months. Progress does not. The toolsmith.

Engineering 274

Engineering Programming Language Hospitality Programming 274

The Pragmatic Engineer

MARCH 3, 2023

On 22 February 2023, Google announced its coding competitions are coming to an end: The visual that accompanied the announcement of the end of Google’s coding competitions. Code Jam: competitive programming. Hash Code: team programming. Google Code Jam I/O for Women: algorithmic programming.

Coding 176

Coding IT Software Engineer Software Engineering 176

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Engineering at Meta

APRIL 6, 2023

Buck2, our new open source, large-scale build system , is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1.

Building 145

Building Systems Java Coding 145

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Tweag

FEBRUARY 15, 2023

Provide those dependencies such that the build system can find them. Run the build system. However, there are some problems with step (4): The build is not properly isolated: it can (in theory) fetch arbitrary things from the network, access arbitrary files on the filesystem, and even modify other packages. The solution to this?

Project 144

Project Programming Language Building Systems 144

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. fix the code # fix code 7.

Systems 141

Systems Building Java Programming Language 141

Data Engineering Podcast

FEBRUARY 18, 2024

Multiple open source projects and vendors have been working together to make this vision a reality. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” years ago, and it became the leading AI coding assistant almost overnight.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform. Backend code I wrote and pushed to prod took down Amazon.com for several hours. and hand-rolled C -code. At the time, this approach was our best effort to deliver code on the nascent web. 

Engineering 302

Engineering Bytes Cloud Computing AWS 302

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Other infrastructure:  Primarily AWS ( S3 for cloud object storage, Parameter Store for hierarchical storage, Elastic Container Service ( ECS ) for container deployment and orchestration) The team manages AWS via infrastructure-as-a-code Pulumi. Tech stack.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

dbt Developer Hub

APRIL 19, 2023

Don’t repeat yourself** : Dimensions can be easily re-used with other fact tables to avoid duplication of effort and code logic. Part 1: Setup dbt project and database ​ Step 1: Install project dependencies ​ Before you can get started: You must have either DuckDB or PostgreSQL installed. For example: select * from sales.

Building 145

Building PostgreSQL BI Database 145

The Pragmatic Engineer

NOVEMBER 26, 2024

model was released, which has superior code generation compared to ChatGPT. To get a sense of how preferences of developers might have shifted, I asked: “If you're a dev: what is your favorite coding editor with GenAI features (that help your work)? An AI-first code editor, rapidly gaining popularity.

Software Engineer 290

Software Engineer Software Engineering Engineering Coding 290

Engineering at Meta

DECEMBER 6, 2023

This is the most significant milestone yet for this project, which began in earnest after Mark Zuckerberg outlined his vision for it in 2019. Throughout the project, we have consulted with a diverse range of external parties to ensure that we’re making the right set of tradeoffs.

Building 145

Building Designing Consulting Accessible 145

Data Engineering Podcast

JULY 23, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. In Hex you can use SQL, Python, R, and no-code visualization together to explore, transform, and model data.

Building 130

Building Machine Learning SQL Python 130

The Pragmatic Engineer

SEPTEMBER 22, 2023

Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of school and build things, founded by venture capitalist, Peter Thiel). Bun has other contributors, but Jared writes the lion’s share of code. Bun has no such constraint.

Programming Language 255

Programming Language Project Coding Engineering 255

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How have the recent breakthroughs in large language models (LLMs) improved your ability to build features in Zenlytic? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Data Engineering Podcast

AUGUST 27, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 130

Database Building PostgreSQL BI 130

Lyft Engineering

JUNE 28, 2023

On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. Shortly after we built it, it was utilized by another pod within our team to build a Real-time Anomaly Detection product. The code for a real-time ML application should be written once and run everywhere.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Data Engineering Podcast

NOVEMBER 13, 2022

Summary Building data products is an undertaking that has historically required substantial investments of time and talent. In this episode he explains the design of the platform and how it builds on agile development principles to help you focus on delivering value. Can you describe what AgileData is and the story behind it?

Building 130

Building Metadata MongoDB MySQL 130

The Pragmatic Engineer

OCTOBER 10, 2023

He’s solved interesting engineering challenges along the way, too – like building observability for Amazon’s EC2 offering, and being one of the first engineers on Uber’s observability platform. From learning to code in Australia, to working in Silicon Valley How did I learn to code?

Software Engineer 268

Software Engineer Software Engineering Architecture Media 268

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. What are the organizational patterns that are required to support effective workflows for building data generation capabilities?

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 29, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 189

Database BI SQL Machine Learning 189

Knowledge Hut

APRIL 16, 2024

Over several decades, projects have been initiated or undertaken due to market demands, business needs, at the behest of customer request, technological advancements and to comply with regulatory requirements. We will look at some of the key project management methods followed in today’s world. Know more about the project description.

Project 98

Project Management Programming Designing 98

The Pragmatic Engineer

OCTOBER 10, 2024

What makes Wordpress popular is a mix of factors: Restrictive license : licensed as  GPLv2 , which allows for commercial use, and for modifying the code; as long as the license remains in place.   Forks gaining momentum that has the code of Wordpress, but uses a different name could also be in the books. Vendors diversifying.

Consulting 306

Consulting AWS Engineering Software Engineer 306

Data Engineering Podcast

SEPTEMBER 10, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

BI 208

BI SQL Data Machine Learning 208

The Pragmatic Engineer

SEPTEMBER 19, 2024

Use tech debt payments to get into the flow and stay in it A good reason to add new comments to old code before you change it is to speed up a code review. When it takes me time to learn what code does, writing something down helps me remember what I figured out. Clarifying the code is even better.

Recruitment 176

Recruitment Java Coding Project 176

Engineering at Meta

DECEMBER 19, 2024

Were sharing details about Glean , Metas open source system for collecting, deriving and working with facts about source code. In August 2021 we open-sourced our code indexing system Glean. Glean collects information about source code and provides it to developer tools through an efficient and flexible query language.

Coding 55

Coding Programming Language SQL Programming 55

The Pragmatic Engineer

MAY 1, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” years ago, and it became the leading AI coding assistant almost overnight.

Software Engineer 200

Software Engineer Software Engineering Programming Language Media 200

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Gone are the months of complex data wrangling and the constraints of no-code SaaS tools.

Cloud 108

Cloud Building Insurance Data Engineering 108

Christophe Blefari

MARCH 1, 2023

dbt Labs also develop dbt Cloud which is a cloud product that hosts and runs dbt Core projects. a dbt project — a dbt project is a folder that contains all the dbt objects needed to work. You can initialise a project with the CLI command: dbt init. In a dbt project you can define YAML file everywhere.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Go to dataengineeringpodcast.com/dagster today to get started.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Your first 30 days are free!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

SEPTEMBER 3, 2023

The dlt project was created to eliminate overhead and bring data integration into your full control as a library component of your overall data system. In this episode Adrian Brudaru explains how it works, the benefits that it provides over other data integration solutions, and how you can start building pipelines today.

Data Integration 162

Data Integration BI SQL Python 162

Knowledge Hut

DECEMBER 22, 2023

For all project managers and those aspiring to become project managers in the future, project management certifications are essential for success together with academic study and real-world experience. We will also update you on how to get Project management certification for beginners.

Certification 98

Certification Project Management Portfolio 98