Data Engineering Podcast

SEPTEMBER 12, 2021

Summary The Cassandra database is one of the first open source options for globally scalable storage systems. The community recently released a new major version that marks a milestone in its maturity and stability as a project and database. Since its introduction in 2008 it has been powering systems at every scale.

Database 100

Database Kafka Metadata Data Storage 100

Cloudera

JANUARY 6, 2021

Python is used extensively among Data Engineers and Data Scientists to solve all sorts of problems from ETL/ELT pipelines to building machine learning models. Apache HBase is an effective data storage system for many workflows but accessing this data specifically through Python can be a struggle.

Machine Learning 133

Machine Learning Data Science Database Building 133

Pinterest Engineering

NOVEMBER 22, 2023

Goku is our in-house time series database providing cost efficient and low latency storage for metrics data. Goku Long Term Storage Architecture Summary and Challenges Figure 9: Flow of data from GokuS to GokuL. short, RocksDB is a key value store that uses a log structure DB engine for storage and retrieval.

Database 111

Database Bytes Kafka Architecture 111

Analytics Vidhya

FEBRUARY 27, 2023

Due to its lack of POSIX conformance, some believe it to be data storage instead. Introduction The Hadoop Distributed File System (HDFS) is a Java-based file system that is Distributed, Scalable, and Portable. HDFS and […] The post Top 10 Hadoop Interview Questions You Must Know appeared first on Analytics Vidhya.

Hadoop 240

Hadoop Java Data Storage Programming 240

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Flume is a tool/service/data ingestion mechanism for gathering, aggregating, and delivering huge amounts of streaming data from diverse sources, such as log files, events, and so on, to centralized data storage. Flume is a tool that is very dependable, distributed, and customizable.

Data Ingestion 223

Data Ingestion Data Storage Hadoop Data 223

Cloudera

SEPTEMBER 9, 2021

The CDP Operational Database ( COD ) builds on the foundation of existing operational database capabilities that were available with Apache HBase and/or Apache Phoenix in legacy CDH and HDP deployments. Cloudera Machine Learning or Cloudera Data Warehouse), to deliver fast data and analytics to downstream components.

Database 95

Database AWS Relational Database Cloud 95

Monte Carlo

AUGUST 15, 2023

These use cases are typically the first and easiest behavior shift for data teams once they enter the cloud. They are: Moving from ETL to ELT to accelerate time-to-insight You can’t just load anything into your on-premise database– especially not if you want a query to return before you hit the weekend.

Data Storage 88

Data Storage Cloud Metadata Machine Learning 88

Analytics Vidhya

FEBRUARY 2, 2023

Introduction In this constantly growing era, the volume of data is increasing rapidly, and tons of data points are produced every second. Now, businesses are looking for different types of data storage to store and manage their data effectively.

Data Lake 207

Data Lake Data Warehouse Data Storage Data 207

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Though basic and easy to use, traditional table storage formats struggle to keep up. Track data files within the table along with their column statistics. Contact phData Today!

Architecture 52

Architecture Systems Data Lake Google Cloud 52

The Pragmatic Engineer

JUNE 13, 2023

Agoda co-locates in all data centers, leasing space for its racks and the largest data center consumes about 1 MW of power. It uses Spark for the data platform. For transactional databases, it’s mostly the Microsoft SQL Server, but also other databases like PostgreSQL, ScyllaDB and Couchbase.

Cloud 251

Cloud Database Utilities BI 251

Monte Carlo

OCTOBER 31, 2024

The foundational skills are similar between traditional data engineers and AI data engineers are similar, with AI data engineers more heavily focused on machine learning data infrastructure, AI-specific tools, vector databases, and LLM pipelines. Let’s dive into the tools necessary to become an AI data engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Striim

MARCH 21, 2025

A streaming ETL for Snowflake approach loads data to Snowflake from diverse sources such as transactional databases, security systems logs, and IoT sensors/devices in real time , while simultaneously meeting scalability, latency, security, and reliability requirements.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Knowledge Hut

MARCH 27, 2024

Think of a database as a smart, organized library that stores and manages information efficiently. On the other hand, data structures are like the tools that help organize and arrange data within a computer program. What is a Database? SQL, or structured query language, is widely used for writing and querying data.

Database 52

Database Relational Database Algorithm Data Storage 52

ThoughtSpot

NOVEMBER 5, 2024

This allows users to interact with their data without interruption, regardless of system scale. This article highlights the performance optimizations implemented to initialize Atlas, our in-house Graph database, in less than two minutes. What is metadata? What is Atlas?

Metadata 52

Metadata PostgreSQL Java Database 52

Christophe Blefari

MARCH 1, 2023

This switch has been lead by modern data stack vision. In terms of paradigms before 2012 we were doing ETL because storage was expensive, so it became a requirement to transform data before the data storage—mainly a data warehouse, to have the most optimised data for querying.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Cloudera

NOVEMBER 1, 2023

And so we are thrilled to introduce our latest applied ML prototype (AMP) — a large language model (LLM) chatbot customized with website data using Meta’s Llama2 LLM and Pinecone’s vector database. High-level overview of real-time data ingest with Cloudera DataFlow to Pinecone vector database.

Machine Learning 110

Machine Learning Data Ingestion Database Architecture 110

Christophe Blefari

NOVEMBER 11, 2022

Kovid wrote an article that tries to explain what are the ingredients of a data warehouse. A data warehouse is a piece of technology that acts on 3 ideas: the data modeling, the data storage and processing engine. Slowly, years after years, graph databases time is coming up. And he does it well.

BI 130

BI Data Warehouse Data Database 130

Knowledge Hut

APRIL 25, 2024

Each of these technologies has its own strengths and weaknesses, but all of them can be used to gain insights from large data sets. As organizations continue to generate more and more data, big data technologies will become increasingly essential. Let's explore the technologies available for big data.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Knowledge Hut

MARCH 14, 2024

Do you want a database system that can scale quickly and manage heavy workloads? Should that be the case, Azure SQL Database might be your best bet. Microsoft SQL Server's functionalities are fully included in Azure SQL Database, a cloud-based database service that also offers greater flexibility and scalability.

Database 52

Database SQL Relational Database BI 52

Knowledge Hut

SEPTEMBER 6, 2023

Back when I studied Computer Science in the early 2000s, databases like MS Access and Oracle ruled. The rise of big data and NoSQL changed the game. Systems evolved from simple to complex, and we had to split how we find data from where we store it. What Is a Database? Now, it's different. Let’s begin!

Certification 52

Certification Database MongoDB MySQL 52

Cloudyard

JANUARY 21, 2025

As new data comes in on day 2, we may have additional columns like PHONE and EMAIL. Handling Parquet Data with Schema Evolution Let’s now look at how schema evolution works with Parquet files. Parquet is a columnar storage format, often used for its efficient data storage and retrieval.

Data Schemas 59

Data Schemas Data Pipeline Data Warehouse Data Storage 59

Ascend.io

NOVEMBER 14, 2024

Ensure the provider supports the infrastructure necessary for your data needs, such as managed databases, storage, and data pipeline services. Utilize Cloud-Native Tools: Leverage cloud-native data pipeline tools like Ascend to build and orchestrate scalable workflows.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Christophe Blefari

JANUARY 20, 2023

In between the Hadoop era, the modern data stack and the machine learning revolution everyone—but me—waits for. But, funny, in the end we are still copying data from database to database by using CSVs, like 40 years ago. Don't be surprised if no ones uses data catalogs.

Google Cloud 130

Google Cloud Data Hadoop Machine Learning 130

Knowledge Hut

JANUARY 18, 2024

Report data findings to management Monitor data collection. Data Analyst Data analysts’ job is to create and maintain data systems and databases. Statistical tools are used to interpret data by the data analyst. Roles and Responsibilities Develop, design, and create data models.

Data Science 98

Data Science Business Analyst Data Architect ETL Method 98

Christophe Blefari

SEPTEMBER 25, 2023

Mainly Motherduck is the company providing DuckDB as a Cloud product but they are not developing DuckDB, their product is quite young but works like expected: with a simple string you can get an analytical cloud database that just works and that can be instantly replaced by a local one if needed. Providing more control over data storage.

Data 130

Data Data Warehouse Data Storage Cloud 130

Christophe Blefari

SEPTEMBER 25, 2023

Mainly Motherduck is the company providing DuckDB as a Cloud product but they are not developing DuckDB, their product is quite young but works like expected: with a simple string you can get an analytical cloud database that just works and that can be instantly replaced by a local one if needed. Providing more control over data storage.

Data 130

Data Data Warehouse Data Storage Cloud 130

Striim

SEPTEMBER 11, 2024

The ingestion layer supports multiple data types and formats, including: Batch Data: Data collected and processed in discrete chunks, typically from static sources such as databases or logs. Data storage Data storage follows. How will we connect to the data sources?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Data Engineering Podcast

AUGUST 17, 2020

Summary Most databases are designed to work with textual data, with some special purpose engines that support domain specific formats. TileDB is a data engine that was built to support every type of data by using multi-dimensional arrays as the foundational primitive. How is the built in data versioning implemented?

Data Engineering 100

Data Engineering Data Engineer Engineering Database Design 100

Knowledge Hut

MARCH 7, 2024

AWS provides more than 200 fully featured services which include storage, database, and computing. Some prominent cloud services offered by Alibaba Cloud include database storage, large-scale computing, network visualization, elastic computing, big data analytics, and management services.

Cloud Computing 98

Cloud Computing Amazon Web Services Cloud Google Cloud 98

Christophe Blefari

JUNE 21, 2024

Both companies have added Data and AI to their slogan, Snowflake used to be The Data Cloud and now they're The AI Data Cloud. Native CDC for Postgres and MySQL — Snowflake will be able to connect to Postgres and MySQL to natively move data from your databases to the warehouse.

Metadata 147

Metadata Data Warehouse BI MySQL 147

Christophe Blefari

JUNE 16, 2023

The ultimate SQL guide — After the last canva on data interviews, here's a canva to learn SQL. From databases introduction to SQL writing. It covers simple SELECT and advanced concepts. This is neat. Malloy's Near Term Roadmap — I've shared recently Malloy demo , which was awesome. but I missed it).

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

FEBRUARY 27, 2022

If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription TimescaleDB, from your friends at Timescale, is the leading open-source relational database with support for time-series data. Time-series data is time stamped so you can measure how a system is changing.

Designing 100

Designing Metadata Data Lake Relational Database 100

Data Engineering Podcast

JANUARY 13, 2020

This requires a new class of data storage which can accomodate that demand without having to rearchitect your system at each level of growth. YugabyteDB is an open source database designed to support planet scale workloads with high data density and full ACID compliance. A growing trend in database engines (e.g.

SQL 100

SQL PostgreSQL MongoDB Database 100

Data Engineering Podcast

AUGUST 14, 2021

Are you spending too much of your engineering resources on creating database views, configuring database permissions, and manually granting and revoking access to sensitive data? Satori has built the first DataSecOps Platform that streamlines data access and security.

Unstructured Data 100

Unstructured Data Machine Learning Data Lake SQL 100

Knowledge Hut

APRIL 25, 2024

Here are some things that you should learn: Recursion Bubble sort Selection sort Binary Search Insertion Sort Databases and Cache To build a high-performance system, programmers need to rely on the cache. In addition, it is required in a database to keep track of the users' responses. to manage DBMS.

Programming Language 98

Programming Language Java Algorithm MySQL 98

Pinterest Engineering

SEPTEMBER 9, 2024

Goku is our in-house time series database that provides cost efficient and low latency storage for metrics data. The Observability team has a tool that uses this data, applies some heuristics on top (not in the scope of this document), and determines metrics which should beblocked.

Database 44

Database Bytes Kafka Software Engineer 44

Hevo

SEPTEMBER 11, 2024

Nowadays, the term is used for petabytes or even exabytes of data (1024 Petabytes), close to trillions of records from billions of people. In this fast-moving landscape, the key to making a difference is picking up the correct data storage solution for your business. […]

Data Lake 40

Data Lake Data Warehouse Database Data Storage 40