Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Your host is Tobias Macey and today I'm welcoming back Gleb Mezhanskiy to talk about how to reconcile data in database environments Interview Introduction How did you get involved in the area of data management?

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. This was the core of your recent re-write of the InfluxDB engine.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

DECEMBER 3, 2023

Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Systems 130

Systems Designing Data Lake SQL 130

Precisely

JANUARY 9, 2025

These are all big questions about the accessibility, quality, and governance of data being used by AI solutions today. The simple idea was, hey how can we get more value from the transactional data in our operational systems spanning finance, sales, customer relationship management, and other siloed functions.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 88

Data Lake Cloud Storage Metadata Data Warehouse 88

Engineering at Meta

JANUARY 22, 2025

It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems. It enhances the traceability of data flows within systems, ultimately empowering developers to swiftly implement privacy controls and create innovative products. Hack, C++, Python, etc.)

Data Warehouse 109

Data Warehouse SQL Programming Language Data 109

Data Engineering Podcast

MAY 18, 2024

Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions. Support Data Engineering Podcast Summary The purpose of business intelligence systems is to allow anyone in the business to access and decode data to help them make informed decisions.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

The Pragmatic Engineer

OCTOBER 17, 2024

The startup was able to start operations thanks to getting access to an EU grant called NGI Search grant. The current database includes 2,000 server types in 130 regions and 340 zones. Results are stored in git and their database, together with benchmarking metadata. Each benchmarking task is evaluated sequentially.

Cloud 314

Cloud AWS Metadata Cloud Computing 314

The Pragmatic Engineer

APRIL 11, 2024

From Sella’s status page : “Following the installation of an update to the operating system and related firmware which led to an unstable situation. The changes messed up all major databases in some unexpected way. Still, I’m puzzled by how long the system has been down.

Banking 255

Banking Utilities Database Engineering 255

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Pinterest Engineering

JANUARY 4, 2024

In 2020, anticipating the growing needs of the business and to simplify our storage offerings, we decided to consolidate our different key-value systems in the company into a single unified service called KVStore. Additionally, the last section explains how this new database supports a key platform in the product.

Database 98

Database Building Datasets Relational Database 98

Analytics Vidhya

FEBRUARY 6, 2023

Introduction HDFS (Hadoop Distributed File System) is not a traditional database but a distributed file system designed to store and process big data. It provides high-throughput access to data and is optimized for […] The post A Dive into the Basics of Big Data Storage with HDFS appeared first on Analytics Vidhya.

Data Storage 187

Data Storage Big Data Hadoop Datasets 187

Striim

JANUARY 30, 2025

Todays organizations have access to more data than ever before, and consequently are faced with the challenge of determining how to transform this tremendous stream of real-time information into actionable insights. Encryption, access controls, and regulatory compliance (HIPAA, GDPR, etc.) patient records or geolocation data).

Systems 52

Systems Management Hospitality Healthcare 52

Snowflake

NOVEMBER 26, 2024

A consolidated data system to accommodate a big(ger) WHOOP When a company experiences exponential growth over a short period, it’s easy for its data foundation to feel a bit like it was built on the fly. This blog post is the second in a three-part series on migrations. million in cost savings annually.

Data Warehouse 103

Data Warehouse Cloud PostgreSQL Hadoop 103

The Pragmatic Engineer

SEPTEMBER 19, 2024

In the early 90’s, DOS programs like the ones my company made had its own Text UI screen rendering system. This rendering system was easy for me to understand, even on day one. Our rendering system was very memory inefficient, but that could be fixed. By doing so, I got to see every screen of the system.

Recruitment 279

Recruitment Java Coding Project 279

Snowflake

NOVEMBER 11, 2024

Unify transactional and analytical workloads in Snowflake for greater simplicity Many businesses must maintain two separate databases: one to handle transactional workloads and another for analytical workloads. Sensitive data can have enormous value but is oftentimes locked down due to privacy requirements.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105

Pinterest Engineering

NOVEMBER 18, 2024

Change Data Capture (CDC) is a crucial technology that enables organizations to efficiently track and capture changes in their databases. In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. What is Change Data Capture?

Kafka 50

Kafka MySQL Database Software Engineer 50

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Engineering at Meta

DECEMBER 19, 2024

Were sharing details about Glean , Metas open source system for collecting, deriving and working with facts about source code. In this blog post well talk about why a system like Glean is important, explain the rationale for Gleans design, and run through some of the ways were using Glean to supercharge our developer tooling at Meta.

Coding 77

Coding Programming Language SQL Programming 77

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. Then you begin researching database objects and find a couple of views, but there are some inconsistencies between them so you do not know which one to use. Your first step might be to locate the orders. Does it sound familiar?

Systems 98

Systems Raw Data Metadata Data Cleanse 98

The Pragmatic Engineer

MAY 11, 2023

A quick summary of these technologies: Prometheus : a time series database. A very popular open-source solution for systems and services monitoring. A fast and open-source column-oriented database management system, which is a popular choice for log management. It evaluates rules and can trigger alerts.

AWS 347

AWS Software Engineer Software Engineering Google Cloud 347

ThoughtSpot

NOVEMBER 5, 2024

ThoughtSpot prioritizes the high availability and minimal downtime of our systems to ensure a seamless user experience. In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. What is Atlas?

Metadata 52

Metadata PostgreSQL Java Database 52

Towards Data Science

FEBRUARY 9, 2024

This involves getting data from an API and storing it in a PostgreSQL database. In the second phase, we’ll develop an application that uses a language model to interact with this database. The second article, which will come later, will delve into creating agents using tools like LangChain to communicate with external databases.

Kafka 98

Kafka Data Engineer Data Engineering PostgreSQL 98

Snowflake

NOVEMBER 11, 2024

Optimize performance and cost with a broader range of model options Cortex AI provides easy access to industry-leading models via LLM functions or REST APIs, enabling you to focus on driving generative AI innovations. We offer a broad selection of models in various sizes, context window lengths and language supports.

Unstructured Data 97

Unstructured Data SQL AWS Healthcare 97

The Pragmatic Engineer

APRIL 6, 2023

Our hope is that making salary ranges more accessible on Comprehensive.io For AI,  we’ve built a system to efficiently use GPT-4 for this purpose, including auto-crafting prompts and performing pre and post-processing. on the backend, and Postgres for database storage.” ” How does Comprehensive.io

Software Engineer 217

Software Engineer Software Engineering Datasets Database 217

Data Engineering Weekly

OCTOBER 27, 2024

Astasia Myers: The three components of the unstructured data stack LLMs and vector databases significantly improved the ability to process and understand unstructured data. I never thought of PDF as a self-contained document database, but that seems a reality that we can’t deny. What are you waiting for?

Data Engineer 57

Data Engineer Data Engineering Engineering Unstructured Data 57

Snowflake

FEBRUARY 13, 2025

Furthermore, most vendors require valuable time and resources for cluster spin-up and spin-down, disruptive upgrades, code refactoring or even migrations to new editions to access features such as serverless capabilities and performance improvements.

Management 82

Management Government Cloud Unstructured Data 82

The Pragmatic Engineer

JUNE 13, 2023

For transactional databases, it’s mostly the Microsoft SQL Server, but also other databases like PostgreSQL, ScyllaDB and Couchbase. queries per second as total load, spread across its managed database-as-a-service (DBAAS.) It uses Spark for the data platform. At peak load, Agoda sees around 7.5M

Cloud 232

Cloud Database Utilities BI 232

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Can you describe what RisingWave is and the story behind it?

SQL 173

SQL Data Lake High Quality Data Machine Learning 173

DataKitchen

NOVEMBER 5, 2024

This architecture is valuable for organizations dealing with large volumes of diverse data sources, where maintaining accuracy and accessibility at every stage is a priority. This foundational layer is a repository for various data types, from transaction logs and sensor data to social media feeds and system logs.

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

Data Engineering Podcast

APRIL 7, 2024

Different roles and tasks in the business need their own ways to access and analyze the data in the organization. In order to enable this use case, while maintaining a single point of access, the semantic layer has evolved as a technological solution to the problem. dbt, BI, warehouse marts, etc.)

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Weekly

AUGUST 6, 2024

CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems. However, managing data consistency across microservices can be challenging.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

Towards Data Science

DECEMBER 10, 2024

TL;DR Take advantage of old school database tricks, like ENUM data types, and column constraints. Some positives (Microsoft Access comes to mind), but some are questionable at best, such as traditional data design principles and data quality and validation at ingestion. Lets get toit! Generate data lineage with one small Pythonscript.

PostgreSQL 75

PostgreSQL Data Python SQL 75

Data Engineering Weekly

JANUARY 15, 2025

The answer lies in unstructured data processing—a field that powers modern artificial intelligence (AI) systems. To address these challenges, AI Data Engineers have emerged as key players, designing scalable data workflows that fuel the next generation of AI systems. Experience with vector databases (e.g.,

Data Engineer 62

Data Engineer Data Engineering Unstructured Data Engineering 62

Striim

OCTOBER 31, 2024

A nonprofit educational healthcare organization is faced with the challenge of modernizing its critical systems while ensuring uninterrupted access to essential services. However, while the SIS migration was a significant step forward, the institution’s on-premise SQL Server systems remained vital.

Education 52

Education Healthcare Data Integration Cloud 52

Ascend.io

NOVEMBER 14, 2024

Ensure the provider supports the infrastructure necessary for your data needs, such as managed databases, storage, and data pipeline services. Leverage Built-In Partitioning Features: Use built-in features provided by databases like Snowflake or Databricks to automatically partition large datasets.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Data Engineering Podcast

JUNE 9, 2024

To address this shortcoming Datorios created an observability platform for Flink that brings visibility to the internals of this popular stream processing system. How have the requirements of generative AI shifted the demand for streaming data systems? What role does Flink play in the architecture of generative AI systems?

Process 147

Process Data Lake High Quality Data Machine Learning 147