Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. Check out the agenda and register at Neo4j.com/NODES.

Building 162

Building BI SQL Machine Learning 162

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

The Pragmatic Engineer

JUNE 1, 2023

for the simulation engine Go on the backend PostgreSQL for the data layer React and TypeScript on the frontend Prometheus and Grafana for monitoring and observability And if you were wondering how all of this was built, Juraj documented his process in an incredible, 34-part blog series. You can read this here. Serving a web page.

Education 363

Education Project PostgreSQL Software Engineer 363

Analytics Vidhya

SEPTEMBER 12, 2024

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge.

Data Pipeline 144

Data Pipeline Building Management Data 144

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform. Backend code I wrote and pushed to prod took down Amazon.com for several hours. and hand-rolled C -code. At the time, this approach was our best effort to deliver code on the nascent web. 

Engineering 302

Engineering Bytes Cloud Computing AWS 302

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” years ago, and it became the leading AI coding assistant almost overnight.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Other infrastructure:  Primarily AWS ( S3 for cloud object storage, Parameter Store for hierarchical storage, Elastic Container Service ( ECS ) for container deployment and orchestration) The team manages AWS via infrastructure-as-a-code Pulumi. Tech stack.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

dbt Developer Hub

APRIL 19, 2023

Close alignment with actual business processes : Business processes and metrics are modeled and calculated as part of dimensional modeling. Part 2: Identify the business process ​ Now that you’ve set up the dbt project, database, and have taken a peek at the schema, it’s time for you to identify the business process.

Building 145

Building PostgreSQL BI Database 145

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What was your process for defining the problem you are focused on solving and the method to achieve that outcome? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Engineering at Meta

DECEMBER 6, 2023

Bringing E2EE to Messenger has been a complex process, with every feature and product goal revealing further challenges that required careful consideration. The post Building end-to-end security for Messenger appeared first on Engineering at Meta.

Building 145

Building Designing Consulting Accessible 145

Data Engineering Podcast

JULY 23, 2023

Summary Real-time data processing has steadily been gaining adoption due to advances in the accessibility of the technologies involved. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Building 130

Building Machine Learning SQL Python 130

Snowflake

JUNE 5, 2024

Enterprises can utilize gen AI to extract more value from their data and build conversational interfaces for customer and employee applications. Addressing a lack of in-house AI expertise and simplifying AI processes can make adoption easier. Generative AI (gen AI) can be transformative for businesses. That’s where Snowflake comes in.

Coding 93

Coding Building Management Government 93

Knowledge Hut

OCTOBER 29, 2023

designing an algorithm to improve the efficiency of hospital processes. Source Code: Hospital Management System 2. Source Code: Weather Forecast App 3. Once you have a dataset, you will need to process it and transform it into a format that can be displayed in your app. Source Code: News Feed App 4.

Computer Science 111

Computer Science Coding Project Hospitality 111

Data Engineering Podcast

AUGUST 27, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 130

Database Building PostgreSQL BI 130

Lyft Engineering

JUNE 28, 2023

While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort. On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. register_feature(feature_definition).add_sink(feature_sink)

Machine Learning 117

Machine Learning Building Metadata Kafka 117

Christophe Blefari

MARCH 15, 2024

Building Meta’s GenAI infrastructure — 2x 24k GPU clusters and it's growing. Arroyo, a stream-processing platform, rebuilt their engine using DataFusion. Coding data pipelines is faster than renting connector catalogs — This is something I've always believed. I'm speechless. This is Croissant.

Metadata 272

Metadata Datasets Data Data Warehouse 272

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Engineering at Meta

JANUARY 29, 2024

Slow build times and inefficiencies in packaging and distributing execution files were costing our ML/AI engineers a significant amount of time while working on our training stack. Our goal in improving build speed was to minimize the need for extensive rebuilding.

Machine Learning 97

Machine Learning Building Python Utilities 97

Data Engineering Podcast

SEPTEMBER 10, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Datafold : ![Datafold]([link]

BI 208

BI SQL Data Machine Learning 208

Engineering at Meta

DECEMBER 19, 2024

Were sharing details about Glean , Metas open source system for collecting, deriving and working with facts about source code. In August 2021 we open-sourced our code indexing system Glean. Glean collects information about source code and provides it to developer tools through an efficient and flexible query language.

Coding 55

Coding Programming Language SQL Programming 55

Snowflake

NOVEMBER 1, 2023

Snowflake has invested heavily in extending the Data Cloud to AI/ML workloads, starting in 2021 with the introduction of Snowpark , the set of libraries and runtimes in Snowflake that securely deploy and process Python and other popular programming languages.

Building 111

Building Python SQL Programming Language 111

The Pragmatic Engineer

MAY 1, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” years ago, and it became the leading AI coding assistant almost overnight.

Software Engineer 200

Software Engineer Software Engineering Programming Language Media 200

Data Engineering Podcast

OCTOBER 29, 2023

In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Database 189

Database BI SQL Machine Learning 189

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today! Your first 30 days are free!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Gone are the months of complex data wrangling and the constraints of no-code SaaS tools.

Cloud 108

Cloud Building Insurance Data Engineering 108

Engineering at Meta

NOVEMBER 15, 2023

Meta is building for the future of AI at every level – from hardware like MTIA v1 , Meta’s first-generation AI inference accelerator to publicly released models like Llama 2 , Meta’s next-generation large language model, as well as new generative AI (GenAI) tools like Code Llama.

Building 103

Building Engineering Transportation Software Engineer 103

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. The services and systems need to be kept up to date, but so does the code that controls their behavior. Search for "Code Commentst" in your podcast player or go to dataengineeringpodcast.com/codecomments today to subscribe.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Go to dataengineeringpodcast.com/dagster today to get started.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

SEPTEMBER 3, 2023

In this episode Adrian Brudaru explains how it works, the benefits that it provides over other data integration solutions, and how you can start building pipelines today. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. in the commercial space.

Data Integration 162

Data Integration BI SQL Python 162

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

OCTOBER 8, 2023

Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Insurance 162

Insurance BI SQL Machine Learning 162

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Your first 30 days are free!

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

SEPTEMBER 24, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

SQL 147

SQL BI Machine Learning Python 147

Data Engineering Weekly

MAY 3, 2024

However, modern Notebooks like Databricks seamlessly integrate with Git to build pull requests and code review processes. Code Reusability The Notebook code is visible within its scope. Dependency management is also bound to the individual scope of the Notebook and prevents production-grade code quality.

Programming Language 119

Programming Language ETL Tools Data Pipeline Coding 119

Tweag

NOVEMBER 20, 2024

A lot of improvements have happened after the initial merge, including but not limited to: Many, many bugfixes in the code generator and runtime, witnessed by the full GHC testsuite for the wasm backend in upstream GHC CI pipelines. Show me the code! $ Now let’s run some code. nix shell gitlab:ghc/ghc-wasm-meta?host

Coding 137

Coding Programming Project Process 137

Data Engineering Podcast

OCTOBER 22, 2023

In this episode Ranjith Raghunath shares his thoughts on how to build a strategy for the development, delivery, and evolution of data products. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

BI 162

BI SQL Machine Learning Programming Language 162