Confessions of a Data Guy

JANUARY 8, 2025

You know, for all the hoards of content, books, and videos produced in the “Data Space” over the last few years, famous or others, it seems I find there are volumes of information on the pieces and parts of working in Data. appeared first on Confessions of a Data Guy.

Data Storage 124

Data Storage Data Pipeline Data IT 124

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Flume is a tool/service/data ingestion mechanism for gathering, aggregating, and delivering huge amounts of streaming data from diverse sources, such as log files, events, and so on, to centralized data storage. Flume is a tool that is very dependable, distributed, and customizable.

Data Ingestion 223

Data Ingestion Data Storage Hadoop Data 223

KDnuggets

AUGUST 23, 2024

Learn how to use Docker volumes to ensure data persistence when working with Docker.

Data Storage 76

Data Storage Data Data Engineer Data Engineering 76

Analytics Vidhya

FEBRUARY 2, 2023

Introduction In this constantly growing era, the volume of data is increasing rapidly, and tons of data points are produced every second. Now, businesses are looking for different types of data storage to store and manage their data effectively.

Data Lake 207

Data Lake Data Warehouse Data Storage Data 207

Monte Carlo

NOVEMBER 12, 2024

When you click on a show in Netflix, you’re setting off a chain of data-driven processes behind the scenes to create a personalized and smooth viewing experience. As soon as you click, data about your choice flows into a global Kafka queue, which Flink then uses to help power Netflix’s recommendation engine.

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

Monte Carlo

AUGUST 15, 2023

Imagine you’ve been building houses with a hammer and nails for most of your career, and I gave you a nail gun. But instead of pressing it to the wood and pulling the trigger, you turn it sideways and hit the nail with the gun as if it were a hammer.

Data Storage 88

Data Storage Cloud Metadata Machine Learning 88

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 139

Data Lake Data Warehouse Data Storage Data 139

Start Data Engineering

SEPTEMBER 4, 2024

Key parts of data systems: 2.1. Data flow design 2.3. Data processing design 2.5. Data storage design 2.7. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools. Introduction 2. Requirements 2.2. Conclusion 1.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Storage 130

Start Data Engineering

MAY 9, 2024

Analyze usage and optimize table data storage 3.2.1. Snowflake pricing and settings inheritance model 3. Strategies to reduce Snowflake cost 3.1. Quick wins by changing settings 3.1.1. Update warehouse settings 3.2. Identify expensive queries and optimize them 3.2.1.1. Identify expensive queries with query_history 3.2.1.2.

Data Storage 130

Data Storage Accessible Accessibility Management 130

KDnuggets

JULY 11, 2022

Learn about the modern tools for data orchestration, data storage, analytical engineering, batch processing, and data streaming.

Data Engineering 137

Data Engineering Data Engineer Engineering Data Storage 137

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Though basic and easy to use, traditional table storage formats struggle to keep up. Track data files within the table along with their column statistics. Contact phData Today!

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Start Data Engineering

OCTOBER 22, 2021

Data modeling 2.1.1. Data storage 2.2. Data transformation 2.2.1. Data pipeline 2.4. Data analytics 3. Introduction 2. SQL skills 2.1. Gathering requirements 2.1.2. Exploration 2.1.3. Modeling 2.1.4. Transformation types 2.2.1.1. Narrow transformations 2.2.1.2. Wide transformations 2.2.2. Practice 4.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

Simon Späti

NOVEMBER 28, 2018

In my recent blog, I researched OLAP technologies, for this post I chose some open-source technologies and used them together to build a full data architecture for a Data Warehouse system. I went with Apache Druid for data storage, Apache Superset for querying and Apache Airflow as a task orchestrator.

Data Warehouse 130

Data Warehouse Data Storage Data Architecture Architecture 130

Seattle Data Guy

DECEMBER 19, 2022

Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Cloudera

APRIL 22, 2022

The focus has also been hugely centred on compute rather than data storage and analysis. In reality, enterprises need their data and compute to occur in multiple locations, and to be used across multiple time frames — from real time closed-loop actions, to analysis of long-term archived data.

Data Solutions 134

Data Solutions Amazon Web Services Data Storage Google Cloud 134

KDnuggets

OCTOBER 22, 2019

For full-stack data science mastery, you must understand data management along with all the bells and whistles of machine learning. This high-level overview is a road map for the history and current state of the expansive options for data storage and infrastructure solutions.

Data Management 123

Data Management Management Data Storage Machine Learning 123

Monte Carlo

OCTOBER 31, 2024

There are dozens of data engineering tools available on the market, so familiarity with a wide variety of these can increase your attractiveness as an AI data engineering candidate. Data Storage Solutions As we all know, data can be stored in a variety of ways.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Christophe Blefari

MARCH 1, 2023

This switch has been lead by modern data stack vision. In terms of paradigms before 2012 we were doing ETL because storage was expensive, so it became a requirement to transform data before the data storage—mainly a data warehouse, to have the most optimised data for querying.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Engineering at Meta

JUNE 12, 2024

Storage We need efficient data-storage solutions to store the vast amounts of data used in model training. This involves investing in high-capacity and high-speed storage technologies and developing new data-storage solutions for specific workloads.

Algorithm 129

Algorithm Data Storage Technology Building 129

Snowflake

NOVEMBER 29, 2023

For example, the data storage systems and processing pipelines that capture information from genomic sequencing instruments are very different from those that capture the clinical characteristics of a patient from a site. A conceptual architecture illustrating this is shown in Figure 3.

Metadata 117

Metadata Healthcare Medical Data Storage 117

phData: Data Engineering

FEBRUARY 21, 2025

Prior to making a decision, an organization must consider the Total Cost of Ownership (TCO) for each potential data warehousing solution. On the other hand, cloud data warehouses can scale seamlessly. Vertical scaling refers to the increase in capability of existing computational resources, including CPU, RAM, or storage capacity.

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Data Engineering Weekly

MARCH 2, 2025

[link] Sneha Ghantasala: Slow Reads for S3 Files in Pandas & How to Optimize it DeepSeek’s Fire-Flyer File System (3FS) re-triggers the importance of an optimized file system for efficient data processing.

Data Engineering 63

Data Engineering Data Engineer Engineering Datasets 63

Edureka

APRIL 22, 2025

Sustainable Data Practices These services could then make it possible for a more environmentally friendly way of handling data to be created and spread, leading to more environmentally friendly data practices.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

Engineering at Meta

MARCH 12, 2024

Storage Storage plays an important role in AI training, and yet is one of the least talked-about aspects. As the GenAI training jobs become more multimodal over time, consuming large amounts of image, video, and text data, the need for data storage grows rapidly.

Building 145

Building Portfolio Utilities Data Storage 145

Knowledge Hut

APRIL 25, 2024

Each of these technologies has its own strengths and weaknesses, but all of them can be used to gain insights from large data sets. As organizations continue to generate more and more data, big data technologies will become increasingly essential. Let's explore the technologies available for big data.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Cloudyard

JANUARY 21, 2025

Handling Parquet Data with Schema Evolution Let’s now look at how schema evolution works with Parquet files. Parquet is a columnar storage format, often used for its efficient data storage and retrieval. We create a table Accessory_parquet and load data from the Parquet file Accessory_day1.parquet

Data Schemas 59

Data Schemas Data Pipeline Data Warehouse Data Storage 59

Striim

MARCH 21, 2025

In-flight transformation also enables a simplified and scalable data architecture that has many related benefits including: Minimizing ETL workloads by performing transformations while data is in motion Optimizing data storage by filtering out unnecessary data Enabling end-to-end recoverability and full resiliency without needing to handle many different (..)

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Cloudera

OCTOBER 1, 2020

Such a status has yet to be granted and without which, data transfers between the UK and the EU will not be lawfully permitted post-December 31st 2020. Without an agreed legislative route to allow data storage and processing in the US and EU, the UK Government will be left with one option; storage and processing within the UK only.

Government 127

Government Cloud Data Storage Architecture 127

Christophe Blefari

SEPTEMBER 25, 2023

I'd say that Iceberg (or table formats) are probably one of the technology that will incrementally change for the better the way we write data pipelines. Providing more control over data storage. Yet I think Iceberg is not yet ready to be widely used ( Python write support still missing, you need Spark).

Data 130

Data Data Warehouse Data Storage Cloud 130

Christophe Blefari

SEPTEMBER 25, 2023

I'd say that Iceberg (or table formats) are probably one of the technology that will incrementally change for the better the way we write data pipelines. Providing more control over data storage. Yet I think Iceberg is not yet ready to be widely used ( Python write support still missing, you need Spark).

Data 130

Data Data Warehouse Data Storage Cloud 130

Monte Carlo

NOVEMBER 21, 2024

This approach is fantastic when you’re not quite sure how you’ll need to use the data later, or when different teams might need to transform it in different ways. It’s more flexible than ETL and works great with the low cost of modern data storage.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

ThoughtSpot

NOVEMBER 5, 2024

It stores all the metadata created within a ThoughtSpot instance to enable efficient querying, retrieval, and management of data objects. While Atlas operates as an in-memory graph database for speed and performance, it uses PostgreSQL as its persistent storage layer to ensure durability and long-term data storage.

Metadata 52

Metadata PostgreSQL Java Database 52

Knowledge Hut

MARCH 7, 2024

Cloud providers can offer you access to the infrastructures such as database services, servers, networks, data management , and data storage. It includes resources such as software, servers, databases, data storage, and networking. Software Cloud service providers offer ready-to-use applications to businesses.

Cloud Computing 98

Cloud Computing Amazon Web Services Cloud Google Cloud 98

Cloudera

MAY 9, 2024

Hybrid Horses for Courses: The Right Cloud for AI from Pilot to Production at Scale Later, on May 14 at 12:40 pm BST , hear from Mark Samson, one of Cloudera’s solutions engineering directors, on whether a data center or cloud deployment is best for your organization’s data platform and architecture.

Banking 102

Banking Data Storage Data Analytics Cloud 102

Data Engineering Weekly

FEBRUARY 2, 2025

The CDC approach addresses challenges like time travel, data validation, performance, and cost by replicating operational data to an AWS S3-based Iceberg Data Lake. The new system automates validation, reduces operational costs by 6x, decreases data storage needs by 1024x, and improves data pipeline performance by 40%.

Data Engineering 58

Data Engineering Data Engineer Engineering Data Lake 58

Cloudera

NOVEMBER 1, 2023

Managing the data that represents organizational knowledge is easy for any developer and does not require exhaustive cycles of data science work. Utilizing Pinecone for vector data storage over an in-house open-source vector store can be a prudent choice for organizations.

Machine Learning 110

Machine Learning Data Ingestion Database Architecture 110