Data Engineering Podcast

APRIL 14, 2024

Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. Can you describe what constitutes a NoSQL database? If you were to start from scratch today, what database would you build?

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

MAY 26, 2024

Summary Any software system that survives long enough will require some form of migration or evolution. When that system is responsible for the data layer the process becomes more challenging. As you have gone through successive migration projects, how has that influenced the ways that you think about architecting data systems?

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Your host is Tobias Macey and today I'm welcoming back Gleb Mezhanskiy to talk about how to reconcile data in database environments Interview Introduction How did you get involved in the area of data management?

Database 147

Database Data Lake High Quality Data Data Workflow 147

KDnuggets

SEPTEMBER 12, 2023

This blog discusses vector databases, specifically pinecone vector databases. A vector database is a type of database that stores data as mathematical vectors, which represent features or attributes. These vectors have multiple dimensions, capturing complex data relationships.

Database 132

Database Machine Learning Data Analysis Systems 132

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. These systems are built on open standards and offer immense analytical and transactional processing flexibility. These formats are transforming how organizations manage large datasets.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Podcast

DECEMBER 3, 2023

Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. This was the core of your recent re-write of the InfluxDB engine.

Database 162

Database Technology Data Lake High Quality Data 162

Simon Späti

MARCH 28, 2025

The database landscape has reached 394 ranked systems across multiple categoriesrelational, document, key-value, graph, search engine, time series, and the rapidly emerging vector databases. What fundamental differences exist between AI-focused vector databases and analytical vector engines like DuckDB or DataFusion?

Technology 130

Technology PostgreSQL MySQL Database 130

Jesse Anderson

FEBRUARY 11, 2025

Semih is a researcher and entrepreneur with a background in distributed systems and databases. He then pursued his doctoral studies at Stanford University, delving into the complexities of database systems. Dont forget to subscribe to my YouTube channel to get the latest on Unapologetically Technical!

Computer Science 130

Computer Science Database Design Software Engineering Software Engineer 130

The Pragmatic Engineer

NOVEMBER 21, 2023

If you had a continuous deployment system up and running around 2010, you were ahead of the pack: but today it’s considered strange if your team would not have this for things like web applications.  We dabbled in network engineering, database management, and system administration. and hand-rolled C -code.

Engineering 345

Engineering Bytes Cloud Computing AWS 345

databricks

DECEMBER 21, 2023

We are excited to announce that Gartner has recognized Databricks as a Leader for a third consecutive year in the 2023 Gartner® Magic.

Database 145

Database Systems Cloud Management 145

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Database 130

Database Building PostgreSQL BI 130

The Pragmatic Engineer

OCTOBER 17, 2024

The current database includes 2,000 server types in 130 regions and 340 zones. Results are stored in git and their database, together with benchmarking metadata. Databases:  SQLite files used to publish data Duck DB to query these files in the public APIs Cockroach DB : used to collect and store historical data.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Cassandra is a NoSQL database management system that is open-source and distributed. It is meant to handle massive volumes of data across many commodity servers while maintaining high availability with no single point of failure.

NoSQL 259

NoSQL Database Project Systems 259

Simon Späti

FEBRUARY 26, 2025

Traditionally, answering this question would require expensive GIS (Geographic Information Systems) software or complex database setups. ” This seemingly simple question requires analyzing competitor locations, population density, traffic patterns, and demographicsall spatial data.

Database 130

Database Data Engineer Data Engineering Accessibility 130

Christophe Blefari

JANUARY 11, 2025

AI companies are aiming for the moon—AGI—promising it will arrive once OpenAI develops a system capable of generating at least $100 billion in profits. Meaning: a YAML configuration system for ingestion and transformations, and now, visualisation with BI-as-code. Meanwhile, the AI landscape remains unpredictable.

Data 130

Data Data Warehouse Coding Programming Language 130

Edureka

APRIL 10, 2025

When you hear the term System Hacking, it might bring to mind shadowy figures behind computer screens and high-stakes cyber heists. In this blog, we’ll explore the definition, purpose, process, and methods of prevention related to system hacking, offering a detailed overview to help demystify the concept.

Systems 40

Systems Education Banking Accessibility 40

Engineering at Meta

JANUARY 22, 2025

It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems. It enhances the traceability of data flows within systems, ultimately empowering developers to swiftly implement privacy controls and create innovative products. Hack, C++, Python, etc.)

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

databricks

DECEMBER 23, 2024

We are excited to announce that Gartner has recognized Databricks as a Leader for a fourth consecutive year in the 2024 Gartner Magic.

Database 110

Database Systems Cloud Management 110

Analytics Vidhya

MARCH 1, 2023

Introduction Cassandra is an Apache-developed free and open-source distributed NoSQL database management system. It manages huge volumes of data across many commodity servers, ensures fault tolerance with the swift transfer of data, and provides high availability with no single point of failure.

NoSQL 230

NoSQL Java Database Big Data 230

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

Analytics Vidhya

FEBRUARY 22, 2023

Introduction Data replication is also known as database replication, which is copying data to ensure that all information remains consistent across all data resources in real-time. data replication is like a safety net that keeps your information safe from disappearing or falling through the cracks. In most cases, data alters.

Database 276

Database Data NoSQL Datasets 276

Monte Carlo

MARCH 28, 2025

Table of Contents Understanding How Data + AI Can Break Data System Code Model Data + AI observability must cover inputs and outputs it is all or nothing Understanding How Data + AI Can Break Data + AI applications are complex. But code takes on new weight in the data + AI system.

Coding 52

Coding Systems Data Pipeline ETL Tools 52

Striim

JANUARY 30, 2025

Systems must be capable of handling high-velocity data without bottlenecks. Striim enables: Real-Time Data Integration Striims distributed, in-memory streaming architecture ingests data from transactional databases, IoT sensors, and application logs in real time. As you can see, theres a lot to consider in adopting real-time AI.

Systems 52

Systems Management Hospitality Healthcare 52

The Pragmatic Engineer

JUNE 1, 2023

Juraj included system monitoring parts which monitor the server’s capacity he runs the app on: The monitoring page on the Rides app And it doesn’t end here. Juraj created a systems design explainer on how he built this project, and the technologies used: The systems design diagram for the Rides application The app uses: Node.js

Education 364

Education Project PostgreSQL Software Engineering 364

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. The changes messed up all major databases in some unexpected way. Still, I’m puzzled by how long the system has been down.

Banking 271

Banking Utilities Database Engineering 271

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 297

Recruitment Java Coding Project 297

Pinterest Engineering

NOVEMBER 22, 2023

Goku is our in-house time series database providing cost efficient and low latency storage for metrics data. As defined generally, a shard is a logical subset of data used for distributing data across distributed systems. To learn more about data points, you can look at the Time Series Data Model in this post.

Database 111

Database Bytes Kafka Architecture 111

Pinterest Engineering

NOVEMBER 18, 2024

Change Data Capture (CDC) is a crucial technology that enables organizations to efficiently track and capture changes in their databases. In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. What is Change Data Capture?

Kafka 50

Kafka MySQL Database Software Engineering 50

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It is a core component of the Apache Hadoop ecosystem and allows for storing and processing large datasets across multiple commodity servers.

Data Storage 187

Data Storage Big Data Hadoop Datasets 187

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Engineering at Meta

JANUARY 21, 2025

Because there are so many different things happening in these systems powered by so many different technologies. However, Strobelight has several safeguards in place to prevent users from causing performance degradation for the targeted workloads and retention issues for the databases Strobelight writes to.

Technology 114

Technology Metadata Utilities Engineering 114

Snowflake

NOVEMBER 26, 2024

A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. This blog post is the second in a three-part series on migrations.

Data Warehouse 105

Data Warehouse Cloud PostgreSQL Hadoop 105

Seattle Data Guy

MAY 8, 2023

If you’re relying on your OLTP system to provide analytics, you might be in for a surprise. While it can work initially, these systems aren’t designed to handle complex queries. … Read more The post OLTP Vs OLAP – What Is The Difference appeared first on Seattle Data Guy.

MongoDB 208

MongoDB SQL Database Designing 208

Precisely

JANUARY 9, 2025

The simple idea was, hey how can we get more value from the transactional data in our operational systems spanning finance, sales, customer relationship management, and other siloed functions. There was no easy way to consolidate and analyze this data to more effectively manage our business. But simply moving the data wasnt enough.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Snowflake

APRIL 9, 2025

KAWA Analytics Digital transformation is an admirable goal, but legacy systems and inefficient processes hold back many companies efforts. It connects structured and unstructured databases across sources and uses a no-code UI or Python for advanced and predictive analytics.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

Snowflake

JANUARY 27, 2025

Experienced engineers in the database space, they wrote a lot of the code that is still powering Snowflake today. Additionally, the infrastructure supporting our systems was unreliable under heavy load, requiring manual retries and frustrating developers with lost productivity. Lets start from the beginning.

Programming Language 93

Programming Language Coding Cloud Systems 93