Data Engineering Podcast

APRIL 14, 2024

Summary Databases come in a variety of formats for different use cases. The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Confluent

NOVEMBER 2, 2021

What will the next important category of databases look like? For decades, relational databases were the undisputed home of data. They powered everything: from websites to analytics, from customer data […].

Database 121

Database Systems Relational Database Data 121

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Data lakes are notoriously complex.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. In the era of the cloud most developers rely on hosted services to manage their databases, but what if you are a cloud service? It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Database 130

Database Building PostgreSQL BI 130

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Knowledge Hut

MARCH 27, 2024

Think of a database as a smart, organized library that stores and manages information efficiently. On the other hand, data structures are like the tools that help organize and arrange data within a computer program. What is a Database? SQL, or structured query language, is widely used for writing and querying data.

Database 52

Database Relational Database Algorithm PostgreSQL 52

Pinterest Engineering

JANUARY 4, 2024

In 2020, anticipating the growing needs of the business and to simplify our storage offerings, we decided to consolidate our different key-value systems in the company into a single unified service called KVStore. Additionally, the last section explains how this new database supports a key platform in the product.

Database 93

Database Building Datasets Relational Database 93

Data Engineering Podcast

DECEMBER 25, 2022

Summary Encryption and security are critical elements in data analytics and machine learning applications. We have well developed protocols and practices around data that is at rest and in motion, but security around data in use is still severely lacking. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

KDnuggets

SEPTEMBER 12, 2023

This blog discusses vector databases, specifically pinecone vector databases. A vector database is a type of database that stores data as mathematical vectors, which represent features or attributes. These vectors have multiple dimensions, capturing complex data relationships.

Database 104

Database Machine Learning Data Analysis Systems 104

Data Engineering Podcast

AUGUST 6, 2021

Summary Every organization needs to be able to use data to answer questions about their business. The trouble is that the data is usually spread across a wide and shifting array of systems, from databases to dashboards. Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Database 100

Database SQL BI Business Intelligence 100

Hevo

DECEMBER 26, 2023

Databases are the cornerstone of almost all business projects. As a result, organizations should focus on designing superior databases to meet the objectives of projects without losing direction. Consequently, Database Schema Design Examples and the Database Schema Example have […]

Designing 98

Designing Database Project Systems 98

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. This generalisation makes their data models complex and cryptic and require domain expertise. Searching for data Imagine being a data engineer/analyst tasked with identifying the top-selling products within your company.

Systems 78

Systems Raw Data Metadata Data Cleanse 78

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. We’re proud to be recognized for the data management and data analytics innovations we have delivered in the new Cloudera Data Platform (CDP). This is unique.

Database 91

Database Cloud Systems Management 91

Cloudera

FEBRUARY 8, 2022

Cloudera has been recognized as a Visionary in 2021 Gartner® Magic Quadrant for Cloud Database Management Systems (DBMS) and for the first time, evaluated CDP Operational Database (COD) against the 12 critical capabilities for Operational Databases. Evolutionary schema is supported.

Database 67

Database Systems Cloud Management 67

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning. Missing data? Stale dashboards?

Database 100

Database Data Lake BI Business Intelligence 100

Hevo

SEPTEMBER 28, 2023

Row store and Column store databases are two different approaches to retrieve and store data from Database Management Systems (DBMS). Both databases are appropriate for many use cases, and each has unique qualities.

Database 52

Database Systems Process Management 52

Towards Data Science

FEBRUARY 9, 2024

The first phase focuses on building a data pipeline. This involves getting data from an API and storing it in a PostgreSQL database. In the second phase, we’ll develop an application that uses a language model to interact with this database. Using these data engineering tools firsthand is beneficial.

Kafka 72

Kafka Data Engineer Data Engineering PostgreSQL 72

Striim

JULY 30, 2024

Ensuring system resilience is critical for maintaining a competitive edge in today’s data-driven world. As businesses rely on real-time data to fuel decision-making, it’s essential that their systems can withstand disruptions and maintain functionality. Check out this blog.) Here’s how it works.

Systems 52

Systems Healthcare Utilities Data Integration 52

Knowledge Hut

DECEMBER 28, 2023

The world is producing data at an unprecedented rate. The answer lies within databases. Imagine having all your business data - from customer information to product inventory - stored in one secure location, easily accessible, ready to be queried, updated, and analysed. What are Database Management Tools?

Database 52

Database MySQL SQL Relational Database 52

Data Engineering Podcast

JUNE 22, 2021

Summary The database is the core of any system because it holds the data that drives your entire experience. We spend countless hours designing the data model, updating engine versions, and tuning performance. RudderStack’s smart customer data pipeline is warehouse-first.

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Monte Carlo

MARCH 6, 2024

If you’ve been keeping track of our integrations page you might have noticed Monte Carlo is adding support for different databases faster than a nested loop server crash. By the end of 2023, we added Azure SQL database to the list as well as enterprise databases such as Oracle DB , SAP HANA and Teradata.

Database 52

Database MySQL Data Warehouse Architecture 52

Data Engineering Podcast

MARCH 5, 2022

Summary When you think about selecting a database engine for your project you typically consider options focused on serving multiple concurrent users. Sometimes what you really need is an embedded database that is blazing fast for single user workloads. And don’t forget to thank them for their continued support of this show!

Database 100

Database Data Lake Java Data Engineer 100

Knowledge Hut

APRIL 23, 2024

We live in a data-driven culture where familiarity with databases is crucial. Database management is crucial for businesses of all sizes to guarantee that their data is complete, safe, and easily available when needed. There will likely be a greater need for database specialists' skills in 2024.

Database 52

Database Management Relational Database SQL 52

Data Engineering Podcast

MAY 29, 2022

Summary A large fraction of data engineering work involves moving data from one storage location to another in order to support different access and query patterns. Singlestore aims to cut down on the number of database engines that you need to run so that you can reduce the amount of copying that is required.

Database 100

Database Architecture Data Architecture PostgreSQL 100

Start Data Engineering

JULY 11, 2020

Intro A very common use case in data engineering is to build a ETL system for a data warehouse, to have data loaded in from multiple separate databases to enable data analysts/scientists to be able to run queries on this data, since the source databases are used by your applications and we do not want these analytic queries to affect our application (..)

Systems 130

Systems Designing ETL System Data Warehouse 130

Knowledge Hut

DECEMBER 28, 2023

Automation is important in handling large databases. If your CI/CD pipelines are not extended to the databases, a wide range of technical problems may arise. The collaboration between developers and database executives could be facilitated using certain strategies, which lays the stage for database DevOps.

Database 52

Database Banking Coding Technology 52

Analytics Vidhya

FEBRUARY 22, 2023

Introduction Data replication is also known as database replication, which is copying data to ensure that all information remains consistent across all data resources in real-time. data replication is like a safety net that keeps your information safe from disappearing or falling through the cracks.

Database 269

Database Data NoSQL Datasets 269

Towards Data Science

OCTOBER 12, 2023

Construction engineer investigating his work — Stable diffusion Introduction In our previous publication, From Data Engineering to Prompt Engineering , we demonstrated how to utilize ChatGPT to solve data preparation tasks. In recent decades, data architectures have grown increasingly diverse and complex. This is a fair point.

Data Integration 67

Data Integration Systems Engineering Datasets 67

Start Data Engineering

MAY 30, 2024

Introduction Data is stored on disk and processed in memory Running the code Run on Codespaces Run on your laptop Using python REPL Python basics Python is used for extracting data from sources, transforming it, & loading it into a destination [Extract & Load] Read and write data to any system [Transform] Process data in Python or instruct (..)

Python 148

Python Data Engineer Data Engineering Engineering 148

Knowledge Hut

DECEMBER 28, 2023

The information system is a very vast concept that encompasses several aspects like database management, the communication system, various devices, several connections, the internet, collection, organization, and storing data and other information-related applications that are typically used in a business forum.

Systems 52

Systems Telecommunication Technology Certification 52

Data Engineering Podcast

JANUARY 5, 2020

Summary Databases are useful for inspecting the current state of your application, but inspecting the history of that data can get messy without a way to track changes as they happen. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council.

Database 100

Database Kafka PostgreSQL MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Edureka

SEPTEMBER 3, 2024

Moving databases to the cloud can be a really challenging and risky process, and it can also interrupt business processes. This fully managed service makes it easier to migrate databases to the cloud, from on-premises, or from one cloud service to another. What is AWS Database Migration Service?

AWS 40

AWS Database MySQL PostgreSQL 40

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. In order to turn this into a tractable problem one approach is to define and enforce contracts between producers and consumers of data. Atlan is the metadata hub for your data ecosystem.

Systems 100

Systems Metadata Building MongoDB 100

Pinterest Engineering

NOVEMBER 22, 2023

Goku is our in-house time series database providing cost efficient and low latency storage for metrics data. Goku is our in-house time series database providing cost efficient and low latency storage for metrics data. To learn more about data points, you can look at the Time Series Data Model in this post.

Database 108

Database Bytes Kafka Architecture 108

Towards Data Science

DECEMBER 1, 2023

Why static workload is insufficient and what I learned by comparing HNSWLIB and DiskANN using streaming workload Image by DALLE-3 Vector databases are built for high-dimensional vector retrieval. Embeddings are used in many applications like search engines, recommendation systems, and chatbots. O(log n) ) with respect to index size.

Database 52

Database Algorithm Datasets Kafka 52