KDnuggets

NOVEMBER 9, 2021

In this article we are discussing that HDF5 is one of the most popular and reliable formats for non-tabular, numerical data. This article suggests what kind of ML native data format should be to truly serve the needs of modern data scientists. But this format is not optimized for deep learning work.

Deep Learning 121

Deep Learning Data Storage Data Data Management 121

Snowflake

APRIL 2, 2025

Data storage has been evolving, from databases to data warehouses and expansive data lakes, with each architecture responding to different business and data needs. Traditional databases excelled at structured data and transactional workloads but struggled with performance at scale as data volumes grew.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

KDnuggets

FEBRUARY 15, 2022

From Oracle, to NoSQL databases, and beyond, read about data management solutions from the early days of the RBDMS to those supporting AI applications.

Data Storage 108

Data Storage Database NoSQL Data 108

Analytics Vidhya

FEBRUARY 27, 2023

Due to its lack of POSIX conformance, some believe it to be data storage instead. Introduction The Hadoop Distributed File System (HDFS) is a Java-based file system that is Distributed, Scalable, and Portable. HDFS and […] The post Top 10 Hadoop Interview Questions You Must Know appeared first on Analytics Vidhya.

Hadoop 240

Hadoop Java Data Storage Programming 240

KDnuggets

AUGUST 23, 2024

Learn how to use Docker volumes to ensure data persistence when working with Docker.

Data Storage 78

Data Storage Data Data Engineering Data Engineer 78

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Flume is a tool/service/data ingestion mechanism for gathering, aggregating, and delivering huge amounts of streaming data from diverse sources, such as log files, events, and so on, to centralized data storage. Flume is a tool that is very dependable, distributed, and customizable.

Data Ingestion 223

Data Ingestion Data Storage Hadoop Data 223

KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 140

Data Lake Data Warehouse Data Storage Data 140

Monte Carlo

AUGUST 15, 2023

And while this analogy isn’t a perfect encapsulation of how some data teams operate after moving from on-premises to a modern data stack, it’s close. So perhaps a better analogy for these recently migrated data teams would be if I gave you 1,000 nail guns…and then watched you turn them all sideways to hit 1,000 nails at the same time.

Data Storage 88

Data Storage Cloud Metadata Machine Learning 88

KDnuggets

JULY 11, 2022

Learn about the modern tools for data orchestration, data storage, analytical engineering, batch processing, and data streaming.

Data Engineer 138

Data Engineer Data Engineering Engineering Data Storage 138

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. Though basic and easy to use, traditional table storage formats struggle to keep up. Track data files within the table along with their column statistics.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Start Data Engineering

SEPTEMBER 4, 2024

Key parts of data systems: 2.1. Data flow design 2.3. Data processing design 2.5. Data storage design 2.7. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools. Introduction 2. Requirements 2.2. Conclusion 1.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Storage 130

Seattle Data Guy

DECEMBER 19, 2022

By Reseun McClendon Today, your enterprise must effectively collect, store, and integrate data from disparate sources to both provide operational and analytical benefits. Whether its helping increase revenue by finding new customers or reducing costs, all of it starts with data.

Data Lake 130

Data Lake Data Warehouse Database Data Storage 130

Edureka

APRIL 22, 2025

Data is more than simply numbers as we approach 2025; it serves as the foundation for business decision-making in all sectors. However, data alone is insufficient. Data engineering can help with it. It is the force behind seamless data flow, enabling everything from AI-driven automation to real-time analytics.

Data Engineering 52

Data Engineering Data Engineer Engineering Consulting 52

KDnuggets

OCTOBER 22, 2019

For full-stack data science mastery, you must understand data management along with all the bells and whistles of machine learning. This high-level overview is a road map for the history and current state of the expansive options for data storage and infrastructure solutions.

Data Management 123

Data Management Management Data Storage Machine Learning 123

Data Engineering Weekly

MARCH 2, 2025

Annual Report: The State of Apache Airflow® 2025 DataOps on Apache Airflow® is powering the future of business – this report reviews responses from 5,000+ data practitioners to reveal how and what’s coming next. Data Council 2025 is set for April 22-24 in Oakland, CA. DeepSeek’s smallpond Takes on Big Data.

Data Engineering 63

Data Engineering Data Engineer Engineering Datasets 63

Monte Carlo

OCTOBER 31, 2024

The rise of AI and GenAI has brought about the rise of new questions in the data ecosystem – and new roles. One job that has become increasingly popular across enterprise data teams is the role of the AI data engineer. Demand for AI data engineers has grown rapidly in data-driven organizations.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Start Data Engineering

OCTOBER 22, 2021

Data modeling 2.1.1. Data storage 2.2. Data transformation 2.2.1. Data pipeline 2.4. Data analytics 3. Introduction SQL is the bread and butter of data engineering. Introduction 2. SQL skills 2.1. Gathering requirements 2.1.2. Exploration 2.1.3. Modeling 2.1.4. Transformation types 2.2.1.1.

SQL 130

SQL Data Engineering Data Engineer Engineering 130

Cloudera

APRIL 22, 2022

Part of this emphasis extends to helping enterprises deal with their data and overall cloud connectivity as well as local networks. At the same time, operators are also becoming more data- and cloud-centric themselves. Networked data and cloud are not just “edge computing”. This poses both opportunities and challenges.

Data Solutions 134

Data Solutions Amazon Web Services Data Storage Cloud 134

Simon Späti

NOVEMBER 28, 2018

However, this is still not common in the Data Warehouse (DWH) field. In my recent blog, I researched OLAP technologies, for this post I chose some open-source technologies and used them together to build a full data architecture for a Data Warehouse system. Why is this?

Data Warehouse 130

Data Warehouse Data Storage Data Architecture Architecture 130

Data Engineering Weekly

FEBRUARY 2, 2025

Notably, the process includes an RL step to create a specialized reasoning model (R1-Zero) capable of excelling in reasoning tasks without labeled SFT data, highlighting advancements in training methodologies for AI models. It employs a two-tower model approach to learn query and item embeddings from user engagement data.

Data Engineer 58

Data Engineer Data Engineering Engineering Data Lake 58

Ascend.io

NOVEMBER 14, 2024

Why Future-Proofing Your Data Pipelines Matters Data has become the backbone of decision-making in businesses across the globe. The ability to harness and analyze data effectively can make or break a company’s competitive edge. Resilience and adaptability are the cornerstones of a future-proof data pipeline.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Monte Carlo

NOVEMBER 21, 2024

Let’s set the scene: your company collects data, and you need to do something useful with it. Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Christophe Blefari

SEPTEMBER 25, 2023

This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open. Providing more control over data storage.

Data 130

Data Data Warehouse Data Storage Cloud 130

Christophe Blefari

SEPTEMBER 25, 2023

This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open. Providing more control over data storage.

Data 130

Data Data Warehouse Data Storage Cloud 130

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision. Enter the ELT.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Christophe Blefari

JANUARY 20, 2023

Summer in coming ( credits ) Hey, new Friday, new Data News edition. Thank you for every recommendation you do about the blog or the Data News. The current state of data This week Benjamin Rogojan livestreamed an online conference featuring awesome data voices: state of data infra.

Google Cloud 130

Google Cloud Data Hadoop Machine Learning 130

Christophe Blefari

JUNE 16, 2023

I'm now under the Berlin rain with 20° When I write in these conditions I feel like a tortured author writing a depressing novel while actually today I'll speak about the AI Act, Python, SQL and data platforms. Mainly he unit tests macros (the logic) with his framework and test data with soda and dbt contracts.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Christophe Blefari

NOVEMBER 11, 2022

On a social note, today I've joined data-folks Mastodon server, you can follow me there. I'll speak about "How to build the data dream team" Let's jump onto the news. Ingredients of a Data Warehouse Going back to basics. Ian describes how Riot Games uses data and what machine learning means.

BI 130

BI Data Warehouse Data Database 130

Start Data Engineering

MAY 9, 2024

Analyze usage and optimize table data storage 3.2.1. Snowflake pricing and settings inheritance model 3. Strategies to reduce Snowflake cost 3.1. Quick wins by changing settings 3.1.1. Update warehouse settings 3.2. Identify expensive queries and optimize them 3.2.1.1. Identify expensive queries with query_history 3.2.1.2.

Data Storage 130

Data Storage Accessibility Accessible Management 130

Data Engineering Weekly

JUNE 10, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Databricks and Snowflake offer a data warehouse on top of cloud providers like AWS, Google Cloud, and Azure.

Data Engineering 103

Data Engineering Data Engineer Engineering Kafka 103

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Snowflake

SEPTEMBER 25, 2023

Digital advancements such as smart manufacturing and automation through AI, machine learning (ML), robotics, and IoT require a connected value chain ecosystem with a secure, scalable, and flexible data platform. Easy data sharing and collaboration A key feature of Snowflake is the ability to share and collaborate on data.

Manufacturing 111

Manufacturing Cloud Electronics BI 111

Data Engineering Weekly

OCTOBER 11, 2024

The data world is abuzz with speculation about the future of data engineering and the successor to the celebrated modern data stack. While the modern data stack has undeniably revolutionized data management with its cloud-native approach, its complexities and limitations are becoming increasingly apparent.

Manufacturing 110

Manufacturing Transportation Data Warehouse Unstructured Data 110

Monte Carlo

MARCH 3, 2025

The real disruption lies with data + AI. In other words, when organizations combine their first-party data with LLMs to unlock unique insights, automate processes, or accelerate specialized workflows. We saw this with software and application observability; data and data observability; and soon data + AI and data + AI observability.

Unstructured Data 52

Unstructured Data Data Cloud Computing Banking 52

Cloudera

MAY 9, 2024

The Gartner Data and Analytics Summit in London is quickly approaching on May 13 th to 15 th , and the Cloudera team is ready to hit the show floor! With the rapid pace of AI innovations, today’s leaders must connect, share, and learn from each other on how to best leverage AI technology to effectively analyze data to our advantage.

Banking 102

Banking Data Storage Data Analytics Cloud 102

Precisely

OCTOBER 24, 2024

Maintaining data integrity during cloud migration is essential to ensure reliable and high-quality data for better decision-making and future use in advanced applications. Is your business getting ready to migrate your data to the cloud?

Cloud 111

Cloud Data Integration Insurance Data 111

Cloudera

JANUARY 6, 2021

Python is used extensively among Data Engineers and Data Scientists to solve all sorts of problems from ETL/ELT pipelines to building machine learning models. Apache HBase is an effective data storage system for many workflows but accessing this data specifically through Python can be a struggle.

Machine Learning 133

Machine Learning Data Science Database Building 133