Data Engineering Podcast

DECEMBER 25, 2022

Recognizing this shortcoming and the capabilities that could be unlocked by a robust solution Rishabh Poddar helped to create Opaque Systems as an outgrowth of his PhD studies. No more shipping and praying, you can now know exactly what will change in your database!

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. In this blog post we’ll dive into how we tackled one source of quality issues: directly relying on upstream database schemas. Each of these microservices has its own database.

Kafka 52

Kafka ETL Tools BI Database 52

Data Engineering Podcast

JUNE 22, 2021

Summary The database is the core of any system because it holds the data that drives your entire experience. Andy Pavlo researches autonomous database systems, and out of that research he created OtterTune to find the optimal set of parameters to use for your specific workload.

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Cloudera

FEBRUARY 8, 2022

Cloudera has been recognized as a Visionary in 2021 Gartner® Magic Quadrant for Cloud Database Management Systems (DBMS) and for the first time, evaluated CDP Operational Database (COD) against the 12 critical capabilities for Operational Databases. Evolutionary schema is supported.

Database 66

Database Systems Cloud Management 66

Confluent

MAY 9, 2019

Event-first thinking enables us to build a new atomic unit: the event. This model is completely free form, we can build anything provided that we apply mechanical sympathy with the underlying system behavior. Four pillars of event streaming. Building the KPay payment system. Building the KPay payment system.

Kafka 94

Kafka Pipeline-centric Architecture Database-centric 94

Workfall

JUNE 27, 2023

In this blog, we’ll explore how to harness the power of Kafka to streamline event streaming within a microservices architecture and unlock its full potential for building scalable and responsive systems. So if a user requests a piece of information, the request goes to the Database server. Let’s get started!

Kafka 75

Kafka Architecture AWS Transportation 75

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning. Start trusting your data with Monte Carlo today!

Database 100

Database Data Lake BI Business Intelligence 100

Scott Logic

APRIL 22, 2024

I’ve coloured the data entities according to their types and we see there’s a few different patterns like events and state which we’ll discuss in a moment. I’ll use the term message in this blog as a generic term for asynchronous communication that may be state or commands or events.

AWS 52

AWS Metadata Systems Kafka 52

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. By exploring the benefits of using Data Mesh for your event streaming architecture, this article will help you decide if it’s the right solution for your organization.

Architecture 52

Architecture Generalist Government Datasets 52

Data Engineering Podcast

AUGUST 6, 2022

For analytical systems there are decades of investment in data warehouses and various modeling techniques. For machine learning applications relational models require additional processing to be directly useful, which is why there has been a growth in the use of vector databases. Who is the target audience for this database?

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

MAY 25, 2021

The problem is that in order for the models to integrate with external systems their internal state has to be translated into a lower dimension. To eliminate this impedance mismatch Edo Liberty founded Pinecone to build database that works natively with vectors. Can you start by describing what Pinecone is and the story behind it?

Database 100

Database Building Data Warehouse Machine Learning 100

Rockset

MARCH 27, 2023

In this post, we’ll explore the differences between real-time analytics databases and stream processing frameworks. Differing Paradigms Stream processing systems and real-time analytics (RTA) databases are both exploding in popularity. Let’s start with a quick summary of both stream processing and RTA databases.

Database 52

Database Process Scala SQL 52

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? Serverless functions provide a synergistic relationship with event streaming applications; they behave differently with respect to streaming workloads but are both event driven. Event-first FaaS? Event-first FaaS.

Kafka 109

Kafka AWS Architecture Cloud 109

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture. Being event first.

Architecture 92

Architecture Programming Kafka Database-centric 92

Data Engineering Podcast

JULY 6, 2020

Summary There are an increasing number of use cases for real time data, and the systems to power them are becoming more mature. You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data platforms.

Systems 100

Systems Kafka SQL Data Engineer 100

Knowledge Hut

JUNE 20, 2023

A logical consequence of these changes is the way fields like database security research topics and DBMS research have come up as the need of the hour. With new technologies and techniques emerging day-by-day, staying up-to-date with the latest trends in database research topics is crucial.

Database 52

Database Java Education Data Collection 52

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Cloudera

OCTOBER 29, 2021

CDP Operational Database (COD) is an autonomous transactional database powered by Apache HBase and Apache Phoenix. COD is easy-to-provision and self-managing, that means developers can provision a new database instance within minutes and start creating prototypes quickly. You can access COD right from your CDP console.

Database 90

Database Amazon Web Services AWS Cloud 90

The Pragmatic Engineer

OCTOBER 17, 2024

The current database includes 2,000 server types in 130 regions and 340 zones. Results are stored in git and their database, together with benchmarking metadata. Databases:  SQLite files used to publish data Duck DB to query these files in the public APIs Cockroach DB : used to collect and store historical data.

Cloud 273

Cloud AWS Metadata Cloud Computing 273

Cloudera

SEPTEMBER 23, 2020

Access audits are mastered centrally in Apache Ranger which provides comprehensive non-repudiable audit log for every access event to every resource with rich access event metadata such as: IP. Cloudera’s platform can support piping of audit data to HDFS, Kafka, Syslog or to SIEM systems for long-term retention and archival.

Database 84

Database Data Lake Metadata Java 84

Data Engineering Podcast

MARCH 5, 2022

Summary When you think about selecting a database engine for your project you typically consider options focused on serving multiple concurrent users. Sometimes what you really need is an embedded database that is blazing fast for single user workloads. Can you describe what DuckDB is and the story behind it?

Database 100

Database Data Lake Java Data Engineer 100

Data Engineering Podcast

JANUARY 5, 2020

Summary Databases are useful for inspecting the current state of your application, but inspecting the history of that data can get messy without a way to track changes as they happen. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh.

Database 100

Database Kafka PostgreSQL MySQL 100

Data Engineering Podcast

JULY 27, 2020

Summary A majority of the scalable data processing platforms that we rely on are built as distributed systems. Kyle Kingsbury created the Jepsen framework for testing the guarantees of distributed data processing systems and identifying when and why they break. This brings with it a vast number of subtle ways that errors can creep in.

Systems 100

Systems Building Scala Java 100

Data Engineering Podcast

APRIL 28, 2019

However, managing the database layer is still a separate concern. The KubeDB project was created as a way of providing a simple mechanism for running your storage system in the same platform as your application. What are the main challenges associated with running a stateful system on top of Kubernetes?

Database 100

Database PostgreSQL MongoDB MySQL 100

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. The changes messed up all major databases in some unexpected way. Still, I’m puzzled by how long the system has been down.

Banking 217

Banking Utilities Database Engineering 217

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Confluent

MARCH 19, 2019

When should you use an event? I will attempt to articulate in layman’s terms what an event-driven architecture (EDA) is and contrast it with service-oriented architecture (SOA). In essence, this is an attempt to differentiate and/or associate APIs with events. They, in some cases, are more resilient than databases.

Architecture 110

Architecture Telecommunication Transportation Database 110

Data Engineering Podcast

JULY 20, 2020

Summary Wind energy is an important component of an ecologically friendly power system, but there are a number of variables that can affect the overall efficiency of the turbines. Michael Tegtmeier founded Turbit Systems to help operators of wind farms identify and correct problems that contribute to suboptimal power outputs.

Systems 100

Systems Machine Learning Manufacturing Algorithm 100

Knowledge Hut

APRIL 25, 2024

Intrusion detection systems (IDS) are designed to identify suspicious and malicious activity through network traffic. So, let's get to know the meaning of an intrusion detection system and how it works. What Is a n Intrusion Detection System? Why Are Intrusion Detection Systems Important? and how it works.

Systems 52

Systems Recruitment Machine Learning Certification 52

Data Engineering Podcast

APRIL 22, 2019

FaunaDB is a cloud native database built by the engineers behind Twitter’s infrastructure and designed to serve the needs of modern systems. To make it easier for startups to focus on delivering useful features Segment offers a flexible and reliable data infrastructure for your customer analytics and custom events.

Database 100

Database Cloud NoSQL Scala 100

Data Engineering Podcast

FEBRUARY 1, 2021

As the systems that we build continue to grow in scale and complexity the need for reliable and manageable monitoring platforms increases proportionately. In this episode Rob Skillington, CTO of Chronosphere, shares his experiences building metrics systems that provide observability to companies that are operating at extreme scale.

Systems 100

Systems Cloud BI Data Warehouse 100

Knowledge Hut

JULY 24, 2023

Recently, the advent of stream processing has unlocked the door for a new era in database technology. In today’s data-driven world, the future of SQL is entwined with the future of databases and becoming highly significant. According to recent studies, the global database market will grow from USD 63.4

Database 52

Database SQL Process NoSQL 52

Rockset

DECEMBER 19, 2023

From his early days at Quora to leading projects at Facebook and his current venture at Fennel (a real-time feature store for ML), Nikhil has traversed the evolving landscape of machine learning engineering and machine learning infrastructure specifically in the context of recommendation systems.

Systems 52

Systems Machine Learning Deep Learning Media 52

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 87

Systems Metadata Media Kafka 87

Netflix Tech

DECEMBER 14, 2018

In this post we will provide details of the NMDB system architecture beginning with the system requirements?—?these A fundamental requirement for any lasting data system is that it should scale along with the growth of the business applications it wishes to serve. key value stores generally allow storing any data under a key).

Media 96

Media Database Metadata Data Schemas 96

Rockset

OCTOBER 4, 2022

Event streams, such as clickstreams, IoT data and other time series data, are common sources of data into these apps. The broad adoption of Apache Kafka has helped make these event streams more accessible. In this post, we compare two options for real-time analytics on event and CDC streams: Rockset and ClickHouse.

MySQL 52

MySQL Kafka Aggregated Data Architecture 52