Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 111

Data Lake Data Warehouse Cloud Unstructured Data 111

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Introduction to the Data Mesh Architecture and its Required Capabilities. Components of a Data Mesh.

Architecture 124

Architecture Metadata Kafka Government 124

Edureka

MAY 27, 2025

Microsoft Fabric is a next-generation data platform that combines business intelligence, data warehousing, real-time analytics, and data engineering into a single integrated SaaS framework. The architecture of Microsoft Fabric is based on several essential elements that work together to simplify data processes: 1.

Architecture 52

Architecture BI Business Intelligence Raw Data 52

Data Engineering Weekly

JANUARY 12, 2025

With Astro, you can build, run, and observe your data pipelines in one place, ensuring your mission critical data is delivered on time. link] Sponsored: Apache Airflow® Best Practices: Running Airflow at Scale The scalability of Airflow is why data teams at companies like Uber, Ford, and LinkedIn choose it to power their data ops.

Pipeline-centric 57

Pipeline-centric Data Engineering Data Engineer Engineering 57

Data Engineering Podcast

JUNE 17, 2021

Summary Working with unstructured data has typically been a motivation for a data lake. Kirk Marple has spent years working with data systems and the media industry, which inspired him to build a platform for automatically organizing your unstructured assets to make them more valuable. No more scripts, just SQL.

Unstructured Data 100

Unstructured Data Data Warehouse Metadata Media 100

Cloudera

JUNE 7, 2022

We live in a hybrid data world. In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB.

IT 112

IT Unstructured Data Data Architecture Government 112

Data Engineering Podcast

FEBRUARY 3, 2019

Summary Archaeologists collect and create a variety of data as part of their research and exploration. Open Context is a platform for cleaning, curating, and sharing this data. I got frustrated at the lack of comparative data, and I got frustrated at all the work I put into creating data that nobody would likely use.

Digital Media 100

Digital Media Media PostgreSQL Datasets 100

AltexSoft

JUNE 7, 2021

Hadoop and Spark are the two most popular platforms for Big Data processing. They both enable you to deal with huge collections of data no matter its format — from Excel tables to user feedback on websites to images and video files. Which Big Data tasks does Spark solve most effectively? How does it work? cost-effectiveness.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Engineering at Meta

MARCH 18, 2024

Users can query using regular expressions on log lines, arbitrary metadata fields attached to logs, and across log files of hosts and services. Logarithm’s data model Logarithm represents logs as a named log stream of (host-local) time-ordered sequences of immutable unstructured text, corresponding to a single log file. in PyTorch).

Engineering 107

Engineering Metadata Architecture Designing 107

Data Engineering Podcast

JULY 1, 2018

Summary Data is often messy or incomplete, requiring human intervention to make sense of it before being usable as input to machine learning projects. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform.

Machine Learning 100

Machine Learning Metadata Data Preparation Data Collection 100

LinkedIn Engineering

JULY 19, 2023

Co-Authors: Sumedh Sakdeo , Lei Sun , Sushant Raikar , Stanislav Pak , and Abhishek Nath Introduction At LinkedIn, we build and operate an open source data lakehouse deployment to power Analytics and Machine Learning workloads. While functional, our current setup for managing tables is fragmented.

Big Data 78

Big Data Data Management Management Metadata 78

Netflix Tech

OCTOBER 27, 2020

By Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy.

Data Warehouse 103

Data Warehouse Datasets Data Big Data 103

Snowflake

JUNE 4, 2024

Generative AI presents enterprises with the opportunity to extract insights at scale from unstructured data sources, like documents, customer reviews and images. It also presents an opportunity to reimagine every customer and employee interaction with data to be done via conversational applications.

Data Security 116

Data Security Machine Learning Unstructured Data SQL 116

Netflix Tech

MARCH 5, 2019

While our engineering teams have and continue to build solutions to lighten this cognitive load (better guardrails, improved tooling, …), data and its derived products are critical elements to understanding, optimizing and abstracting our infrastructure. In the Reliability space, our data teams focus on two main approaches.

Cloud 75

Cloud Building Amazon Web Services Metadata 75

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value.

Engineering 102

Engineering Raw Data Data Science Machine Learning 102

Cloudera

MAY 17, 2021

Making a decision on a cloud data warehouse is a big deal. Modernizing your data warehousing experience with the cloud means moving from dedicated, on-premises hardware focused on traditional relational analytics on structured data to a modern platform.

Data Warehouse 90

Data Warehouse Cloud Government Metadata 90

Engineering at Meta

FEBRUARY 6, 2024

The script we use to generate DotSlash files injects metadata about the build job that makes it straightforward to trace the provenance of the underlying artifacts. The following is a hypothetical example of a generated DotSlash file for the CodeCompose LSP built from source at a specific commit in clang-opt mode.

Metadata 133

Metadata Coding Building Project 133

RandomTrees

SEPTEMBER 17, 2024

In the realm of big data and AI, managing and securing data assets efficiently is crucial. Databricks addresses this challenge with Unity Catalog, a comprehensive governance solution designed to streamline and secure data management across Databricks workspaces. What is Unity Catalog? Advantages of the Unity Catalog 1.

Data Governance 52

Data Governance Government Metadata Machine Learning 52

Cloudera

JUNE 25, 2021

Many Cloudera customers are making the transition from being completely on-prem to cloud by either backing up their data in the cloud, or running multi-functional analytics on CDP Public cloud in AWS or Azure. The Replication Manager service facilitates both disaster recovery and data migration across different environments.

Cloud 73

Cloud Data Lake Cloud Storage Metadata 73

Cloudera

APRIL 1, 2024

This recognition underscores Cloudera’s commitment to continuous customer innovation and validates our ability to foresee future data and AI trends, and our strategy in shaping the future of data management. Cloudera, a leader in big data analytics, provides a unified Data Platform for data management, AI, and analytics.

Cloud 105

Cloud Unstructured Data Metadata Government 105

Sync Computing

NOVEMBER 7, 2024

Data is central to modern business and society. Depending on what sort of leaky analogy you prefer, data can be the new oil , gold , or even electricity. Of course, even the biggest data sets are worthless, and might even be a liability, if they arent organized properly.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Unstructured Data 52

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

SEPTEMBER 15, 2022

Apache Ozone is a distributed, scalable, and high-performance object store , available with Cloudera Data Platform (CDP), that can scale to billions of objects of varying sizes. Structured data (such as name, date, ID, and so on) will be stored in regular SQL databases like Hive or Impala databases. Diversity of workloads.

Systems 87

Systems Hadoop Metadata Telecommunication 87

Monte Carlo

AUGUST 25, 2023

Modern companies are ingesting, storing, transforming, and leveraging more data to drive more decision-making than ever before. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Netflix Tech

DECEMBER 14, 2018

In the previous blog posts in this series, we introduced the N etflix M edia D ata B ase ( NMDB ) and its salient “Media Document” data model. A fundamental requirement for any lasting data system is that it should scale along with the growth of the business applications it wishes to serve.

Media 99

Media Database Metadata Data Schemas 99

Towards Data Science

JANUARY 17, 2024

Parquet vs ORC vs Avro vs Delta Lake Photo by Viktor Talashuk on Unsplash The big data world is full of various storage systems, heavily influenced by different file formats. These are key in nearly all data pipelines, allowing for efficient data storage and easier querying and information extraction. So let’s get started!

Big Data 97

Big Data Data Data Storage SQL 97

Netflix Tech

OCTOBER 31, 2018

the Media Timeline Data Model In the previous post in this series, we described some important Netflix business needs as well as traits of the media data system?—?called The curious reader might have noticed that a majority of these characteristics relate to properties of the data managed by NMDB.

Media 55

Media Metadata Data MongoDB 55

AltexSoft

NOVEMBER 10, 2021

Well, there’s a new phenomenon in data management that received the name of a data lakehouse. The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse?

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

AltexSoft

AUGUST 22, 2022

The landscape of enterprise data is fragmented. Organizations have data stored in public and private clouds, as well as in various on-premises data repositories. Some would say that it’s not a big deal, however, these mixed environments have resulted in the complexities of managing disjointed data and business processes.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

Ascend.io

FEBRUARY 28, 2024

The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How does AI recalibrate the workload and priorities of data teams? How can data engineers harness the power of AI?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind. How can I use this data? Is this data up-to-date?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Government 52

Data Engineering Weekly

MARCH 24, 2023

Why should we care about Data Creation Process? All Successful Data-Driven organizations have one thing in common; They have a high-quality & efficient data creation process. Data creation is often the differentiator between the success & the failure of a data team. The riders request a new ride.

Engineering 67

Engineering Data Transportation Database 67

Monte Carlo

FEBRUARY 12, 2024

To differentiate and expand the usefulness of these models, organizations must augment them with first-party data – typically via a process called RAG (retrieval augmented generation). Today, this first-party data mostly lives in two types of data repositories. Quality : Is the data itself anomalous?

Database 72

Database Unstructured Data Data Pipeline Metadata 72

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design. It is called Idempotency.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

RandomTrees

SEPTEMBER 6, 2020

Snowflake Overview A data warehouse is a critical part of any business organization. Lot of cloud-based data warehouses are available in the market today, out of which let us focus on Snowflake. Snowflake is an analytical data warehouse that is provided as Software-as-a-Service (SaaS).

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

Towards Data Science

MARCH 5, 2024

Make the most out of your BigQuery usage, burn data rather than money to create real value with some practical techniques. · ? Introduction In the field of data warehousing, there’s a universal truth: managing data can be costly. But let me give you a magical spell to appease the dragon: burn data, not money!

Bytes 97

Bytes Google Cloud Cloud Storage Utilities 97

ProjectPro

FEBRUARY 16, 2023

ETL is a critical component of success for most data engineering teams, and with teams harnessing it with the power of AWS, the stakes are higher than ever. Data Engineers and Data Scientists require efficient methods for managing large databases, which is why centralized data warehouses are in high demand.

AWS 52

AWS Data Management ETL Tools Management 52