Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Transportation Data 110

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Traditional data warehouse platform architecture. Data lake. Data lake architecture example.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

ThoughtSpot

NOVEMBER 5, 2024

In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. Any delays in metadata retrieval can negatively impact user experience, resulting in decreased productivity and satisfaction. What is Atlas?

Metadata 52

Metadata PostgreSQL Java Database 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

SEPTEMBER 23, 2021

One of the innovative ways to address this problem is to build a data hub — a platform that unites all your information sources under a single umbrella. This article explains the main concepts of a data hub, its architecture, and how it differs from data warehouses and data lakes. What is Data Hub?

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52

Snowflake

NOVEMBER 29, 2023

For example, the data storage systems and processing pipelines that capture information from genomic sequencing instruments are very different from those that capture the clinical characteristics of a patient from a site. A conceptual architecture illustrating this is shown in Figure 3.

Metadata 87

Metadata Healthcare Medical Data Storage 87

Netflix Tech

SEPTEMBER 18, 2024

This abstraction simplifies data access, enhances the reliability of our infrastructure, and enables us to support the broad spectrum of use cases that Netflix demands with minimal developer effort. Data Model At its core, the KV abstraction is built around a two-level map architecture. number of chunks).

Bytes 99

Bytes Metadata Database Data 99

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Engineering at Meta

MARCH 6, 2024

Our technical solution builds on Meta’s existing client / server architecture We think the best way to deliver interoperability is through a solution which builds on Meta’s existing client / server architecture [Figure 1]. The proof is constructed by the third-party service cryptographically signing an authentication token.

Media 131

Media Architecture Metadata Data Storage 131

Data Engineering Weekly

OCTOBER 13, 2024

link] Lak Lakshmanan: What goes into bronze, silver, and gold layers of a medallion data architecture? If I understand correctly, the gist of the article is where you position the common data model/ metrics that can be used across the organization. I think these layers are a guiding principle instead of a strict framework.

Data Engineering 52

Data Engineering Data Engineer Engineering PostgreSQL 52

Christophe Blefari

JANUARY 20, 2024

formats — This is a huge part of data engineering. Picking the right format for your data storage. Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). Wrong format often means bad querying performance and user-experience.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Cloudera

SEPTEMBER 15, 2022

Structured data (such as name, date, ID, and so on) will be stored in regular SQL databases like Hive or Impala databases. There are also newer AI/ML applications that need data storage, optimized for unstructured data using developer friendly paradigms like Python Boto API. FILE_SYSTEM_OPTIMIZED Bucket (“FSO”).

Systems 85

Systems Hadoop Metadata Telecommunication 85

Data Engineering Weekly

OCTOBER 6, 2024

Join us on October 10th at 9am PT | 12pm ET as I walk through how data observability platforms like Monte Carlo can help data analytics teams reduce time spent on data quality by 80% or more. Though the blog talks about Superset, every team should strive to adopt a similar framework to manage the lifecycle of their dashboards.

Data Engineering 62

Data Engineering Data Engineer Engineering Data Warehouse 62

Christophe Blefari

JUNE 16, 2023

Change Data Capture (CDC) with PostgreSQL and ClickHouse — This is a nice vendor post about CDC with Kafka as movement layer (using Debezium). The post explains well the architecture you need to make it work. also as a data observability platform, but done differently. It covers simple SELECT and advanced concepts.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

SEPTEMBER 12, 2021

That leaves DataOps reactive to data quality issues and can make your consumers lose confidence in your data. By connecting to your pipeline orchestrator like Apache Airflow and centralizing your end-to-end metadata, Databand.ai lets you identify data quality issues and their root causes from a single dashboard.

Database 100

Database Metadata Kafka Data Storage 100

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Monte Carlo

AUGUST 25, 2023

Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. That’s why it’s essential for teams to choose the right architecture for the storage layer of their data stack. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Monte Carlo

JUNE 20, 2024

But when you’re scaling to become the #1 Data Observability Platform , you’re likely to get more requests than your existing architecture can accommodate. It consisted of three core components: Data connection: the connectivity to resources like Redshift, Snowflake, BigQuery, Databricks and many more (e.g.,

AWS 52

AWS Data Storage Google Cloud Cloud 52

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Towards Data Science

DECEMBER 1, 2023

Technical Overview The project architecture is depicted as follows: Google Calendar -> Fivetran -> Snowflake -> dbt -> Snowflake Dashboard , with GitHub Actions orchestrating the deployment. Storage —  Snowflake Snowflake, a cloud-based data warehouse tailored for analytical needs, will serve as our data storage solution.

Data Engineering 69

Data Engineering Data Engineer Project Engineering 69

Big Data Tools

SEPTEMBER 6, 2021

There are also several changes in KRaft (namely Revise KRaft Metadata Records and Producer ID generation in KRaft mode ), along with many other changes. Unfortunately, the feature that was most awaited (at least by me) – tiered storage – has been postponed for a subsequent release. Support for Scala 2.12 And more files means more time.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

ProjectPro

SEPTEMBER 16, 2021

NoSQL databases are the new-age solutions to distributed unstructured data storage and processing. The speed, scalability, and fail-over safety offered by NoSQL databases are needed in the current times in the wake of Big Data Analytics and Data Science technologies. Hence, writes in Hbase are operation intensive.

NoSQL 52

NoSQL Database Hadoop Big Data 52

ProjectPro

NOVEMBER 7, 2016

Understanding the Hadoop architecture now gets easier! This blog will give you an indepth insight into the architecture of hadoop and its major components- HDFS, YARN, and MapReduce. We will also look at how each component in the Hadoop ecosystem plays a significant role in making Hadoop efficient for big data processing.

Hadoop 40

Hadoop Architecture IT Big Data 40

Cloudera

NOVEMBER 23, 2021

Security and governance policies are set once and applied across all data and workloads. Atlas provides open metadata management and governance capabilities to build a catalog of all assets, and also classify and govern these assets. Apache HBase.

Database 89

Database Non-relational Database NoSQL Government 89

Cloudera

MARCH 22, 2023

Over the past handful of years, systems architecture has evolved from monolithic approaches to applications and platforms that leverage containers, schedulers, lambda functions, and more across heterogeneous infrastructures. And, crucial for a hybrid data platform, it does so across hybrid cloud.

Data Governance 86

Data Governance Government Business Analyst Metadata 86

Databand.ai

JUNE 28, 2023

They’re betting their business on it and that the data pipelines that run it will continue to work. Context is crucial (and often lacking) A major cause of data quality issues and pipeline failures are transformations within those pipelines. Most data architecture today is opaque—you can’t tell what’s happening inside.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

ProjectPro

FEBRUARY 8, 2023

Let us dive deeper into this data integration solution by AWS and understand how and why big data professionals leverage it in their data engineering projects. When Glue receives a trigger, it collects the data, transforms it using code that Glue generates automatically, and then loads it into Amazon S3 or Amazon Redshift.

AWS 98

AWS Scala Metadata Data Lake 98

AltexSoft

JULY 29, 2022

a runtime environment (sandbox) for classic business intelligence (BI), advanced analysis of large volumes of data, predictive maintenance , and data discovery and exploration; a store for raw data; a tool for large-scale data integration ; and. a suitable technology to implement data lake architecture.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

Towards Data Science

FEBRUARY 15, 2023

Concepts, theory, and functionalities of this modern data storage framework Photo by Nick Fewings on Unsplash Introduction I think it’s now perfectly clear to everybody the value data can have. To use a hyped example, models like ChatGPT could only be built on a huge mountain of data, produced and collected over years.

Data Lake 76

Data Lake Data Warehouse Architecture Hadoop 76

Snowflake

APRIL 5, 2023

Traditionally, the dimensional data modeling approach is used to build complex data warehouses, while Data Vaults are used in data warehouses to offer long-term historical data storage while modeling. Why is data modeling important for a data warehouse?

Data Warehouse 52

Data Warehouse Healthcare Unstructured Data Metadata 52

Cloudera

JANUARY 21, 2021

With CDW, as an integrated service of CDP, your line of business gets immediate resources needed for faster application launches and expedited data access, all while protecting the company’s multi-year investment in centralized data management, security, and governance. Architecture overview. Separate storage.

IT 92

IT Data Lake Data Warehouse Cloud Storage 92

RandomTrees

SEPTEMBER 6, 2020

Lot of cloud-based data warehouses are available in the market today, out of which let us focus on Snowflake. Snowflake is an analytical data warehouse that is provided as Software-as-a-Service (SaaS). Built on new SQL database engine, it provides a unique architecture designed for the cloud.

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

AltexSoft

JUNE 7, 2021

The system automatically replicates information to prevent data loss in the case of a node failure. Hadoop architecture, or how the framework works. Master Nodes control and coordinate two key functions of Hadoop: data storage and parallel processing of data. Data storage options.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

Rockset

JUNE 6, 2023

The Challenge: Achieving Compute-Storage Separation without Performance Degradation Traditionally databases have been designed to work off of systems with directly attached storage. This simplifies the system’s architecture and enables high bandwidth, low latency access to the data at query time.

Metadata 52

Metadata Datasets Architecture Algorithm 52

Knowledge Hut

MAY 9, 2024

Data Architect ScyllaDB Data architects play a crucial role in designing an organization's data management framework by assessing data sources and integrating them into a centralized plan. Responsibilities Data architects assess an organization's data sources and design plans for centralized data management.

Data Science 52

Data Science Data Architect Data Mining Programming Language 52

Big Data Tools

SEPTEMBER 6, 2021

There are also several changes in KRaft (namely Revise KRaft Metadata Records and Producer ID generation in KRaft mode ), along with many other changes. Unfortunately, the feature that was most awaited (at least by me) – tiered storage – has been postponed for a subsequent release. Support for Scala 2.12 And more files means more time.

Data Engineering 40

Data Engineering Data Engineer Engineering Big Data Tools 40