Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. Hex brings everything together.

Building 189

Building SQL BI Python 189

Engineering at Meta

MARCH 12, 2024

By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models.

Building 145

Building Portfolio Utilities Data Storage 145

Data Engineering Podcast

JUNE 9, 2024

Summary Streaming data processing enables new categories of data products and analytics. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling. Can you describe how Datorios is used in a day-to-day workflow for someone building streaming applications on Flink?

Process 147

Process Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

OCTOBER 1, 2023

Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows. In this episode Jay Mishra shares his experiences and insights building ETL pipelines with the help of generative AI. Check out the agenda and register at Neo4j.com/NODES.

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Building 147

Building Data Lake High Quality Data Machine Learning 147

databricks

AUGUST 30, 2024

Learn how companies can create repeatable and scalable workflows that enable users to quickly turn GenAI innovation from experimentation to reality.

Process 119

Process Building Data 119

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. When performing research and building prototypes of the projects, what is your process for incorporating user experience into the implementation of the product?

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. We feel your pain. It ends up being anything but that.

Building 147

Building Machine Learning Metadata Data Integration 147

KDnuggets

APRIL 3, 2023

LangChain is a Python library that helps you build GPT-powered applications in minutes. Get started with LangChain by building a simple question-answering app.

Building 160

Building Python Process 160

Analytics Vidhya

SEPTEMBER 12, 2024

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge.

Data Pipeline 144

Data Pipeline Building Management Data 144

dbt Developer Hub

APRIL 19, 2023

Close alignment with actual business processes : Business processes and metrics are modeled and calculated as part of dimensional modeling. Part 2: Identify the business process ​ Now that you’ve set up the dbt project, database, and have taken a peek at the schema, it’s time for you to identify the business process.

Building 145

Building PostgreSQL BI Database 145

Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What was your process for defining the problem you are focused on solving and the method to achieve that outcome? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

The Pragmatic Engineer

APRIL 21, 2024

A refresher on OpenAI, and on Evan Evan: how did you join OpenAI, and end up heading the Applied engineering group – which also builds ChatGPT? I do  not  have a PhD in Machine Learning, and was excited by the idea of building APIs and engineering teams. We do it through a process called  pretraining.

Engineering 351

Engineering Software Engineer Software Engineering Programming 351

Engineering at Meta

DECEMBER 6, 2023

Bringing E2EE to Messenger has been a complex process, with every feature and product goal revealing further challenges that required careful consideration. The post Building end-to-end security for Messenger appeared first on Engineering at Meta.

Building 145

Building Designing Consulting Accessibility 145

Data Engineering Podcast

AUGUST 27, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold You shouldn't have to throw away the database to build with fast-changing data.

Database 130

Database Building PostgreSQL BI 130

databricks

NOVEMBER 26, 2024

Large language models are revolutionizing how we interact with technology by leveraging advanced natural language processing to perform complex tasks. In recent years.

Building 116

Building Technology Process Machine Learning 116

KDnuggets

SEPTEMBER 7, 2023

This article suggests a Microservice approach to separate model integration from business applications and simplify the process. As LLMs continue to evolve, integrating multiple models or switching between them has become increasingly challenging.

Building 141

Building Process 141

Confluent

FEBRUARY 6, 2024

Streamline data processing and unlock analytics to boost productivity and time to market while lowering infrastructure costs. Confluent enables real-time, reliable, scalable, and secure communication between IoT devices, applications, and backend systems.

Building 119

Building Data Data Process Systems 119

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Code and raw data repository:   Version control: GitHub Heavily using GitHub Actions for things like getting warehouse data from vendor APIs, starting cloud servers, running benchmarks, processing results, and cleaning up after tuns. Tech stack.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

The Pragmatic Engineer

JUNE 1, 2023

for the simulation engine Go on the backend PostgreSQL for the data layer React and TypeScript on the frontend Prometheus and Grafana for monitoring and observability And if you were wondering how all of this was built, Juraj documented his process in an incredible, 34-part blog series. You can read this here. ” stage Improve deploys.

Education 363

Education Project PostgreSQL Software Engineer 363

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform.   We didn’t build our applications in neat containers, but in bulky monoliths which commingled business, database, backend, and frontend logic. Our deployments were initially manual. Apache started to log like a maniac.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

DECEMBER 24, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Materialize]([link] You shouldn't have to throw away the database to build with fast-changing data. Data lakes are notoriously complex.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Analytics Vidhya

FEBRUARY 6, 2023

Introduction The demand for data to feed machine learning models, data science research, and time-sensitive insights is higher than ever thus, processing the data becomes complex. To make these processes efficient, data pipelines are necessary.

Amazon Web Services 218

Amazon Web Services Data Pipeline Machine Learning Data Science 218

Christophe Blefari

MARCH 15, 2024

Building Meta’s GenAI infrastructure — 2x 24k GPU clusters and it's growing. Arroyo, a stream-processing platform, rebuilt their engine using DataFusion. The devil is in the details and when it comes to data pipelines there are a lot of details, which often refrain us to buy leading to build (or code).

Metadata 272

Metadata Datasets Data Data Warehouse 272

Analytics Vidhya

MARCH 2, 2023

Introduction Data normalization is the process of building a database according to what is known as a canonical form, where the final product is a relational database with no data redundancy. More specifically, normalization involves organizing data according to attributes assigned as part of a larger data model.

Database 221

Database Relational Database Building Process 221

The Pragmatic Engineer

MARCH 19, 2024

A first, smaller wave of these stories included Magic.dev raising $100M in funding from Nat Friedman (CEO of GitHub from 2018-2021,) and Daniel Gross (cofounder of search engine Cue which Apple acquired in 2013,) to build a “superhuman software engineer.” And COBOL was just one of many attempts.

Software Engineer 276

Software Engineer Software Engineering Programming Language Media 276

The Pragmatic Engineer

AUGUST 22, 2023

Q: “I play video games, and I was wondering how building games compares to more ‘standard’ software development. At Skype, I worked on the team building Skype for the Xbox One console, and several engineers on that team had previously worked in games. To get the full issues, twice a week, subscribe here.

Software Engineer 207

Software Engineer Software Engineering Consulting Entertainment 207

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe a typical workflow for teams that are building on top of RisingWave?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Cloudera

DECEMBER 17, 2024

The introduction of these faster, more powerful networks has triggered an explosion of data, which needs to be processed in real time to meet customer demands. As more data is processed, carriers increasingly need to adopt hybrid cloud architectures to balance different workload demands.

Building 55

Building Telecommunication Data Architecture Architecture 55

Data Engineering Podcast

OCTOBER 29, 2023

In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection. When building analytical systems there are numerous moving parts with intricate dependencies. Datafold : ![Datafold]([link]

Database 189

Database BI SQL Machine Learning 189

The Pragmatic Engineer

APRIL 6, 2023

This created an opportunity to build job sites which collect this data, make it easy to browse, and allow job seekers to apply to jobs paying at or above a certain level. For AI,  we’ve built a system to efficiently use GPT-4 for this purpose, including auto-crafting prompts and performing pre and post-processing.

Software Engineer 194

Software Engineer Software Engineering Datasets Database 194