Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Lyft Engineering

JUNE 28, 2023

On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases. One key component is the Analytics Event Abstraction layer.

Machine Learning 116

Machine Learning Building Metadata Kafka 116

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? Serverless functions provide a synergistic relationship with event streaming applications; they behave differently with respect to streaming workloads but are both event driven. Event-first FaaS? What is FaaS?

Kafka 109

Kafka AWS Architecture Cloud 109

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture. Data evolution.

Architecture 92

Architecture Programming Kafka Database-centric 92

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Four pillars of event streaming. Building the KPay payment system. Putting it all together.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

Snowflake

JUNE 10, 2024

Leveraging Snowflake’s Query History, Event Tables , Alerts and Notifications as the telemetry foundation, Snowflake Trail provides enhanced visibility into data quality, pipelines and applications. Snowflake Trail builds upon the observability foundation already in Snowflake.

Python 105

Python Java Hadoop Coding 105

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Cloudera

JUNE 17, 2022

In this third installment of the Universal Data Distribution blog series, we will take a closer look at how CDF-PC’s new Inbound Connections feature enables universal application connectivity and allows you to build hybrid data pipelines that span the edge, your data center, and one or more public clouds. What are inbound connections?

Data Pipeline 93

Data Pipeline Building Kafka Java 93

Zalando Engineering

NOVEMBER 8, 2023

Postgres Logical Replication at Zalando Builders at Zalando have access to a low-code solution that allows them to declare event streams that source from Postgres databases. At the time of writing, there are hundreds of these Postgres-sourced event streams out in the wild at Zalando. Simple, right?

PostgreSQL 138

PostgreSQL Java Database Bytes 138

Data Engineering Podcast

JULY 27, 2020

In this episode he shares his approach to testing complex systems, the common challenges that are faced by engineers who build them, and why it is important to understand their limitations. Go to dataengineeringpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today!

Systems 100

Systems Building Scala Java 100

Knowledge Hut

MAY 3, 2024

3 Needs re-configuration for Scaling Scales easily by just adding java processes, No reconfiguration required. cache, local space) 8 It supports multiple languages such as Java, Scala, R, and Python. Java is the primary language that Apache Kafka supports. Processing Type Kafka analyses events as they often take place.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Data Engineering Podcast

DECEMBER 30, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Process 100

Process Building Hadoop Java 100

Christophe Blefari

JUNE 21, 2024

you could write the same pipeline in Java, in Scala, in Python, in SQL, etc.—with While I enjoy these events, I prefer avoid flying for ecological reasons, and large gatherings can be challenging for an introvert like me. Databricks successfully built hype during the event, announcements after announcements.

Metadata 147

Metadata Data Warehouse BI Scala 147

Rockset

MARCH 18, 2024

For instance, GPT-4’s cutoff date was April 2023, so it would not be aware of any events or developments happening beyond that point of time. Building a real-time, contextual and trustworthy knowledge base for AI applications revolves around RAG pipelines. What are the challenges building RAG pipelines? What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Confluent

JUNE 11, 2019

For event streaming application developers, it is important to continuously update the streaming pipeline based on the need for changes in the individual applications in the pipeline. It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline.

Kafka 86

Kafka Cloud Java MongoDB 86

Data Engineering Podcast

DECEMBER 22, 2019

To address that need more completely the team at Materialize has created an engine that allows for building queryable views of your data as it is continually updated from the stream of changes being generated by your applications. Between those use cases there are varying levels of support for fast reads on quickly changing data.

SQL 100

SQL Engineering Building Java 100

Striim

MARCH 22, 2023

Before we set up our environment and start building and deploying the image, let’s go through the definitions of both tools. Hashicorp Packer is an open-source Infrastructure-as-Code (IaC) tool that enables you to quickly build and deploy custom images for cloud and on-premises environments. and Java JDK (V1.8.0) What is Striim?

Google Cloud 52

Google Cloud Cloud Amazon Web Services Building 52

Data Engineering Podcast

OCTOBER 22, 2023

In this episode Ranjith Raghunath shares his thoughts on how to build a strategy for the development, delivery, and evolution of data products. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

BI 162

BI SQL Machine Learning Programming Language 162

Netflix Tech

OCTOBER 14, 2021

In the aftermath Ryan Sleevi wrote a really great blog post not just about the issue of this one certificate’s expiry, but the problem that so many TLS implementations have in general with certificate path building. If a client only trusted “AddTrust External CA Root” then this would be fine; that client can build a certificate chain 1 ?

Certification 67

Certification Building Algorithm Java 67

Data Engineering Podcast

OCTOBER 8, 2023

Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Materialize : ![Materialize]([link]

Insurance 162

Insurance BI SQL Machine Learning 162

Confluent

JUNE 18, 2019

Software projects of all sizes and complexities have a common challenge: building a scalable solution for search. Building a resilient and scalable solution is not always easy. It involves many moving parts, from data preparation to building indexing and query pipelines. Building an indexing pipeline at scale with Kafka Connect.

Architecture 97

Architecture Building Kafka Database-centric 97

Zalando Engineering

JUNE 21, 2016

Java programming roles need to cover a lot of ground when it comes to knowledge and processes. We’ve put together a list of essential points that developers should be familiar with when applying for a Java development position. You’ll want to use functional idioms, but don’t overuse them: Java is not a functional language.

Java 40

Java NoSQL Programming Language Kafka 40

Netflix Tech

JULY 22, 2024

To mitigate these risks and assist users to build parameterized workflows, we developed our own customized expression language parser, a simple, secure, and safe expression language (SEL). It leverages the Java Security Manager to restrict access, ensuring a secure and controlled environment for code execution.

Machine Learning 102

Machine Learning Coding Data Pipeline Java 102

Ripple Engineering

MAY 27, 2021

At Ripple , we are moving towards building complex business models out of raw data. We did not have a single tool that would automate the building, compiling, testing and documenting of SQL models, so we had no way to scale the process. Internally, we capture the raw application events in real time in our data lake on BigQuery.  This

Building 52

Building Raw Data SQL Data 52

Knowledge Hut

JUNE 26, 2023

This article is mainly focused on explaining different backend projects for beginners or students, intermediate learners, or those who have mid enough software development experience building large scalable projects. It's good to have some understanding of programming languages like, Java, Python or Node.js.

Portfolio 98

Portfolio Project Food Media 98

Knowledge Hut

SEPTEMBER 24, 2024

This profession can be defined as a skilled computer developer trained to build software, either all on their own or through team effort. Studio 3T Studio 3T, primarily for MongoDB, helps to develop rapid queries, build instant codes, and import and export in several formats. Who is a Software Developer?

Electronics 98

Electronics MongoDB SQL Java 98

Monte Carlo

MAY 31, 2022

We leverage the open source ANTLR parser, which we heavily customized for various dialects of SQL, in a Java-based lambda function to comb through the query logs and generate lineage data. onFailure(funcName, qe, exception)) } } Spline provided a head-start in building Spark lineage. ”)) new SparkLineageInitializer(sparkSession).createListener(isCodelessInit

Data Lake 52

Data Lake Building Scala Metadata 52

Edureka

AUGUST 26, 2024

These are basically a collection of technologies used together to build web applications. Angular is an advanced front-end framework created by Google for building dynamic single-page applications. The great thing about any of these stacks is that they teach important skills necessary for building strong web applications.

MongoDB 97

MongoDB Programming Language MySQL Certification 97

LinkedIn Engineering

SEPTEMBER 20, 2023

When I started my role here, I had to shift to using a different tech stack: Java for the backend and JavaScript framework for the frontend. When I first started out, I came from a biology background, so I was not aware of programming languages and how to translate them into building a technical resume.

Engineering 89

Engineering Programming Language Java Media 89

Rockset

MAY 31, 2019

Live dashboards can help organizations make sense of their event data and understand what's happening in their businesses in real time. In many situations, it is important to be able to take immediate action based on real-time event data, as in the case of limited-time sales in e-commerce or managing contact center service levels.

BI 40

BI Java Data SQL 40

Christophe Blefari

MARCH 24, 2023

I really liked the presentations from Mage and Kestra and even if I was focus on hosting the event it was great to see 2 other visions about the future of orchestration. Tommy, Mage's CEO, said that for the moment they will focus on building the best open-source data pipelines tool. They got enough funding for the next 2 years.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Christophe Blefari

JANUARY 20, 2024

Zuckerberg teased 2024 Meta AI strategy — In a selfie video on Facebook / Instagram Zucky explained that Llama 3 is coming and that Meta is building a massive 600k H100 NVidia GPU infrastructure. Far from trends and the lights, it's actually time to build tools. in pre-seed to build GenAI monitoring applications.

Data 130

Data Python Google Cloud Algorithm 130

Data Engineering Podcast

NOVEMBER 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 130

Data Lake Data Ingestion MongoDB Google Cloud 130

Data Engineering Podcast

APRIL 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 100

Machine Learning Systems Data Lake Java 100

Booking.com Engineering

SEPTEMBER 27, 2022

Introduction: Let me walk you through a real problem solved at Booking.com, which will be a good background to show the original Java code improving, getting smaller and more expressive as it gets more functional. The sorted events list contains all the days that an event (a price starting or ending) happens.

Coding 52

Coding Java Algorithm AWS 52

Netflix Tech

APRIL 12, 2022

GraphQL federation enables domain teams to independently build and operate their own Domain Graph Services (DGS) and, at the same time, connect their domain with other domains in a unified GraphQL schema exposed by a federated gateway. So we need to build an index of a portion of the federated graph. Ours did too!

Engineering 107

Engineering Kafka Java Architecture 107

Data Engineering Podcast

NOVEMBER 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB 130

MongoDB Scala MySQL Machine Learning 130