Data Engineering Podcast

APRIL 16, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What was your process for defining the problem you are focused on solving and the method to achieve that outcome? Who are the target users?

Business Intelligence 147

Business Intelligence Building Data Lake BI 147

Knowledge Hut

APRIL 25, 2023

In this article, we will look at how a scrum master facilitates events such as daily scrum meetings, sprint planning, sprint review, and sprint retrospective meetings. They also support the team in improving their process and collaborating more successfully. Also, we will discuss what qualities they must have to be great facilitators.

Utilities 98

Utilities Project Certification Process 98

dbt Developer Hub

APRIL 19, 2023

Close alignment with actual business processes : Business processes and metrics are modeled and calculated as part of dimensional modeling. Part 2: Identify the business process ​ Now that you’ve set up the dbt project, database, and have taken a peek at the schema, it’s time for you to identify the business process.

Building 145

Building PostgreSQL BI Database 145

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Lyft Engineering

JUNE 28, 2023

While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort. On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft.

Machine Learning 117

Machine Learning Building Metadata Kafka 117

The Pragmatic Engineer

JUNE 1, 2023

for the simulation engine Go on the backend PostgreSQL for the data layer React and TypeScript on the frontend Prometheus and Grafana for monitoring and observability And if you were wondering how all of this was built, Juraj documented his process in an incredible, 34-part blog series. You can read this here. ” stage Improve deploys.

Education 363

Education Project PostgreSQL Software Engineer 363

AltexSoft

AUGUST 25, 2021

And this technology of Natural Language Processing is available to all businesses. Available methods for text processing and which one to choose. Tools you can use to build NLP models. What is Natural Language Processing? Here are some big text processing types and how they can be applied in real life.

Process 139

Process Deep Learning Datasets Machine Learning 139

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Code and raw data repository:   Version control: GitHub Heavily using GitHub Actions for things like getting warehouse data from vendor APIs, starting cloud servers, running benchmarks, processing results, and cleaning up after tuns. Tech stack.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

AltexSoft

MAY 12, 2022

Our brains are constantly processing sounds to give us important information about our environment. Audio analysis is a process of transforming, exploring, and interpreting audio signals recorded by digital devices. It offers a non-invasive type of remote patient monitoring to detect events like falling. What is audio analysis?

Machine Learning 122

Machine Learning Building Deep Learning Datasets 122

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Pipeline 130

Data Pipeline Building MongoDB MySQL 130

Knowledge Hut

JANUARY 28, 2024

The change control process is a crucial aspect of project management intended to manage and regulate changes made to the project plan, schedule, and budget. These change control process steps are planning, analyzing, approval, testing, implementing, and closing. The change request kickstarts the process of change control.

Process 98

Process Project Designing Pharmaceutical 98

Netflix Tech

MARCH 14, 2023

We build creator tooling to enable these colleagues to focus their time and energy on creativity. They may scrub for frames, shots, or scenes of specific characters, locations, objects, events (e.g. We implemented a batch processing system for users to submit their requests and wait for the system to generate the output.

Media 120

Media Building Algorithm Machine Learning 120

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 108

Cloud Building Insurance Data Engineering 108

Pinterest Engineering

JANUARY 4, 2024

In order to build a distributed and replicated service using RocksDB, we built a real time replicator library: Rocksplicator. Maintaining these disparate systems and building common functionality among them was adding a huge overhead to the teams. RocksDB is a single node key value store.

Database 94

Database Building Datasets Relational Database 94

Precisely

DECEMBER 28, 2023

This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. Streaming data pipelines, by extension, offer an architecture capable of handling large volumes of data, accommodating millions of events in near real time. That’s where streaming data pipelines come into play.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

The Pragmatic Engineer

NOVEMBER 21, 2023

  He then worked at the casual games company Zynga, building their in-game advertising platform.   We didn’t build our applications in neat containers, but in bulky monoliths which commingled business, database, backend, and frontend logic. Our deployments were initially manual. Apache started to log like a maniac.

Engineering 302

Engineering Bytes Cloud Computing AWS 302

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Don't miss out on our only event this year!

Database 162

Database Technology Data Lake High Quality Data 162

Snowflake

APRIL 20, 2023

Snowflake users are already taking advantage of LLMs to build really cool apps with integrations to web-hosted LLM APIs using external functions , and using Streamlit as an interactive front end for LLM-powered apps such as AI plagiarism detection , AI assistant , and MathGPT. Join us in Vegas at our Summit to learn more.

Building 120

Building Unstructured Data Government Coding 120

Data Engineering Podcast

MAY 21, 2023

Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Knowledge Hut

APRIL 16, 2024

Going for CSM certification training and knowing how to build a self-organizing team as a Scrum master will help you get trained well. Most Agile-Scrum organizations emphasize on building the self-organizing team - why? How Do You Build a Self-Organizing Team as a Scrum Master?

Building 98

Building Certification Project Management 98

LinkedIn Engineering

JANUARY 19, 2024

This multi-entity handover process involves huge amounts of data updating and cloning. Data consistency, feature reliability, processing scalability, and end-to-end observability are key drivers to ensuring business as usual (zero disruptions) and a cohesive customer experience. Push for eventual success of the request.

Recruitment 88

Recruitment Data Process Process Kafka 88

Data Engineering Podcast

MAY 28, 2023

Summary Building a data team is hard in any circumstance, but at a startup it can be even more challenging. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

The Pragmatic Engineer

OCTOBER 31, 2023

” And an update inside the final hour and a half of the outage: “2:00 PM PDT Many AWS services are now fully recovered and marked Resolved on this event. Lambda is working to process these messages during the next few hours and during this time, we expect to see continued delays in the execution of asynchronous invocations.

AWS 232

AWS Google Cloud Cloud Engineering 232

Confluent

JUNE 27, 2024

You’re not alone—GenAI apps can be challenging to build. Luckily, taking an event-driven approach can make the process more manageable. Struggling to get your GenAI app off the ground? Learn how.

Building 80

Building Process Management 80

Workfall

SEPTEMBER 12, 2023

This makes it a powerful tool for building interactive and dynamic web applications that require instant data exchange between clients and servers, such as chat applications, online gaming, collaborative tools, and more. Its user-friendly API simplifies development, embracing an event-driven structure for custom interactions.

Building 89

Building MongoDB Programming Language Coding 89

The Pragmatic Engineer

MAY 25, 2023

Glassdoor could make the process a lot clearer by publishing a moderation log which details when and why it removed a review. Such a log would build confidence that Glassdoor is a neutral platform which is only enforcing its own terms and conditions, and could validate this. Remember which company has what type of incentives.

Software Engineer 316

Software Engineer Software Engineering AWS Engineering 316

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Don't miss out on our only event this year!

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Netflix Tech

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Process 88

Process Data Pipeline Datasets SQL 88

Data Engineering Podcast

APRIL 9, 2023

Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. In this episode Darren Haken is joined by Tejas Manohar to discuss how Autotrader UK is addressing their customer data needs by building on top of their existing data stack.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year!

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

The Pragmatic Engineer

MARCH 3, 2023

I asked Googlers the reason why these events have been canceled and one thing became clear: most of the program managers who worked on the coding competitions were recently let go in Google’s historic job cuts. In the beginning of February, Google announced the delays in the registration process.

Coding 176

Coding IT Software Engineer Software Engineering 176

Data Engineering Podcast

MARCH 25, 2023

The peril is that building a robust and scalable streaming architecture is always more complicated and error-prone than you think it's going to be. RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code. Don't miss out on their only event this year!

MySQL 130

MySQL Python Architecture Machine Learning 130

Zalando Engineering

NOVEMBER 8, 2023

Postgres Logical Replication at Zalando Builders at Zalando have access to a low-code solution that allows them to declare event streams that source from Postgres databases. At the time of writing, there are hundreds of these Postgres-sourced event streams out in the wild at Zalando. Simple, right?

PostgreSQL 138

PostgreSQL Java Database Bytes 138

Data Engineering Podcast

JUNE 17, 2023

By incorporating column-level lineage in the data modeling process it encourages a more robust and well-informed design. In this episode Satish Jayanthi explores the benefits of incorporating column-aware tooling in the data modeling process. When is column-aware modeling the wrong choice? RudderStack also supports real-time use cases.

Data Lake 130

Data Lake Machine Learning Metadata Data Architecture 130

The Pragmatic Engineer

SEPTEMBER 28, 2023

Willem Spruijt is a software engineer whom I worked on the same team with at Uber in Amsterdam, building payments systems. Going from idea, to adding, to building and shipping this to all customers, is something you rarely see in bigger companies. We cover one out of four topics in today’s subscriber-only The Pulse issue.

Software Engineer 253

Software Engineer Software Engineering Engineering Building 253