Monte Carlo

MAY 28, 2024

A data ingestion architecture is the technical blueprint that ensures that every pulse of your organization’s data ecosystem brings critical information to where it’s needed most. Data Storage : Store validated data in a structured format, facilitating easy access for analysis.

Data Ingestion 52

Data Ingestion Architecture Designing Hadoop 52

Cloudera

JUNE 27, 2023

The way to achieve this balance is by moving to a modern data architecture (MDA) that makes it easier to manage, integrate, and govern large volumes of distributed data. When you deploy a platform that supports MDA you can consolidate other systems, like legacy data mediation and disparate data storage solutions.

Data Architecture 65

Data Architecture Architecture Data Governance Government 65

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Storage layer 3.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Christophe Blefari

FEBRUARY 24, 2023

When it comes to data storage, the real-time ecosystem has also changed a lot in the last few years and a lot of tooling went out to simplify the burden of managing Kafka clusters, Materialize—a real-time platform—detailed their architecture. Hubert tries to explain stuff in the real-time category.

Kafka 130

Kafka Data Lake Data Storage Data 130

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Storage layer 3.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Hevo

DECEMBER 15, 2024

Introduction to Data Architecture Data architecture shows how data is managed, from collection to transformation to distribution and consumption. It tells about how data flows through the data storage systems. Data architecture is an important piece of data management.

Data Architecture 40

Data Architecture Architecture Data Storage Data 40

Databand.ai

AUGUST 30, 2023

DataOps Architecture: 5 Key Components and How to Get Started Ryan Yackel August 30, 2023 What Is DataOps Architecture? DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Engineering at Meta

MARCH 6, 2024

Our technical solution builds on Meta’s existing client / server architecture We think the best way to deliver interoperability is through a solution which builds on Meta’s existing client / server architecture [Figure 1]. The proof is constructed by the third-party service cryptographically signing an authentication token.

Media 135

Media Architecture Metadata Data Storage 135

Engineering at Meta

MARCH 12, 2024

Through careful co-design of the network, software, and model architectures, we have successfully used both RoCE and InfiniBand clusters for large, GenAI workloads (including our ongoing training of Llama 3 on our RoCE cluster) without any network bottlenecks.

Building 145

Building Portfolio Utilities Data Storage 145

Engineering at Meta

JUNE 12, 2024

Storage We need efficient data-storage solutions to store the vast amounts of data used in model training. This involves investing in high-capacity and high-speed storage technologies and developing new data-storage solutions for specific workloads.

Algorithm 143

Algorithm Data Storage Technology Building 143

Cloudera

NOVEMBER 1, 2023

And so we are thrilled to introduce our latest applied ML prototype (AMP) — a large language model (LLM) chatbot customized with website data using Meta’s Llama2 LLM and Pinecone’s vector database. An overview of the RAG architecture with a vector database used to minimize hallucinations in the chatbot application.

Machine Learning 115

Machine Learning Data Ingestion Database Architecture 115

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Cloudera

OCTOBER 1, 2020

Since 2013 the UK Government’s flagship ‘Cloud First’ policy has been at the forefront of enabling departments to shed their legacy IT architecture in order to meaningfully embrace digital transformation. Whilst two of the big three have UK data centres – what happens if they go down?

Government 131

Government Cloud Data Storage Architecture 131

Striim

NOVEMBER 8, 2023

Data Mesh is revolutionizing event streaming architecture by enabling organizations to quickly and easily integrate real-time data, streaming analytics, and more. In this article, we will explore the advantages and limitations of data mesh, while also providing best practices for building and optimizing a data mesh with Striim.

Architecture 52

Architecture Generalist Government Datasets 52

Knowledge Hut

APRIL 25, 2024

Each of these technologies has its own strengths and weaknesses, but all of them can be used to gain insights from large data sets. As organizations continue to generate more and more data, big data technologies will become increasingly essential. Let's explore the technologies available for big data.

Big Data 98

Big Data Technology Hadoop NoSQL 98

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. Benjamin Kennedy, Cloud Solutions Architect at Striim, emphasizes the outcome-driven nature of data pipelines.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. While data vault has many benefits, it is a sophisticated and complex methodology that can present challenges to data quality.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Cloudera

MAY 9, 2024

Hybrid Horses for Courses: The Right Cloud for AI from Pilot to Production at Scale Later, on May 14 at 12:40 pm BST , hear from Mark Samson, one of Cloudera’s solutions engineering directors, on whether a data center or cloud deployment is best for your organization’s data platform and architecture.

Banking 108

Banking Data Storage Data Analytics Cloud 108

Data Engineering Podcast

DECEMBER 31, 2018

Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Tom Kaitchuck about Pravega, an open source data storage platform optimized for persistent streams Interview Introduction How did you get involved in the area of data management?

Lambda Architecture 100

Lambda Architecture Process Data Process Kafka 100

Data Engineering Weekly

DECEMBER 2, 2023

The Current State of the Data Architecture S3 intelligent tiered storage provides a fine balance between the cost and the duration of the data retention. However, the real-time insight on accessing the recent data remains a big challenge. The combination of stream processing + OLAP storage like Pinot.

IT 85

IT BI AWS Kafka 85

Christophe Blefari

JANUARY 20, 2024

formats — This is a huge part of data engineering. Picking the right format for your data storage. This is a new way to describe data platforms with a warehouse at the core where all the company data and KPIs sit. The cherry on the cake here is the Slowly Changing Dimensions — SCDs — concept.

Data Engineer 130

Data Engineer Data Engineering Engineering Hadoop 130

Data Engineering Podcast

JANUARY 28, 2019

What are some of the most common challenges around managing access and authentication for data storage systems? Can you describe the architecture of your system? What strategies have you used to enable interfacing with such a wide variety of storage systems? Can you describe the architecture of your system?

Accessibility 100

Accessibility Accessible Database Management 100

Data Engineering Podcast

SEPTEMBER 12, 2021

What are some of the tools and system architectures that users turn to when building analytical workloads for data stored in Cassandra? The architecture of Cassandra has lent itself well to the cloud native ecosystem that has been growing in recent years. Cassandra is primarily used as a system of record.

Database 100

Database Metadata Kafka Data Storage 100

Cloudera

JANUARY 26, 2022

Modern, real-time businesses require accelerated cycles of innovation that are expensive and difficult to maintain with legacy data platforms. The hybrid cloud’s premise—two data architectures fused together—gives companies options to leverage those solutions and to address decision-making criteria, on a case-by-case basis. .

Cloud 92

Cloud Cloud Computing Hadoop Data Warehouse 92

Cloudera

MARCH 5, 2024

The powerful platform data security and governance layer, Shared Data Experience (SDX) , is a fundamental part of the open data lakehouse, in the data center just as it is in the cloud. Zero Downtime Upgrades Beyond improvements to Iceberg and Ozone, the platform now boasts Zero Downtime Upgrades (ZDU).

Data Lake 92

Data Lake Data Storage Government Kafka 92

Data Engineering Podcast

AUGUST 17, 2020

He also discusses the shifts in database architectures from vertically integrated monoliths to separately deployed layers, and the approach he is taking with TileDB cloud to embed the authorization into the storage engine, while providing a flexible interface for compute. How is the built in data versioning implemented?

Data Engineer 100

Data Engineer Data Engineering Engineering Database Design 100

Christophe Blefari

JANUARY 20, 2023

There is an introduction post about DataHub — when you look at what you have to run to launch a data catalog: 4 components and 4 different data storage. Don't be surprised if no ones uses data catalogs. When I think that some people are saying Airflow is complex to launch.

Google Cloud 130

Google Cloud Data Hadoop Machine Learning 130

Snowflake

JUNE 3, 2024

Either way, they want the freedom to safely use multiple engines on a single copy of data to minimize the storage and compute costs associated with moving data or maintaining multiple copies. Catalogs play a critical role in a multi-engine architecture.

Amazon Web Services 128

Amazon Web Services Google Cloud Data Architect Government 128

Cloudera

NOVEMBER 1, 2022

Translation: Government agencies — especially those under the Department of Defense (DoD) — have use cases that require data storage and analytic workloads to be maintained on premises to retain absolute control of data security, privacy, and cost predictability. . It’s here where the private cloud delivers.

Cloud 95

Cloud Government Cloud Computing Data Architecture 95

DataKitchen

APRIL 28, 2021

Gartner included data fabrics in their top ten trends for data and analytics in 2019. From an industry perspective, the topic of data fabrics is on fire. What is a Data Fabric? Whenever a new technology or architecture gains momentum, vendors hijack it for their own marketing purposes.

Data Pipeline 98

Data Pipeline Data Data Analytics Architecture 98

Data Engineering Podcast

JANUARY 13, 2020

This requires a new class of data storage which can accomodate that demand without having to rearchitect your system at each level of growth. YugabyteDB is an open source database designed to support planet scale workloads with high data density and full ACID compliance. What are the use cases that it is uniquely suited to?

SQL 100

SQL PostgreSQL MongoDB Database Design 100

Knowledge Hut

MARCH 7, 2024

Cloud providers can offer you access to the infrastructures such as database services, servers, networks, data management , and data storage. It includes resources such as software, servers, databases, data storage, and networking. Want to get greater insights into AWS architecture?

Cloud Computing 98

Cloud Computing Amazon Web Services Cloud Google Cloud 98

Christophe Blefari

JUNE 16, 2023

Change Data Capture (CDC) with PostgreSQL and ClickHouse — This is a nice vendor post about CDC with Kafka as movement layer (using Debezium). The post explains well the architecture you need to make it work. A deep dive into graph analytics — Petrica tries and showcases Memgraph in a long-form post. This is neat.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Knowledge Hut

JANUARY 18, 2024

Data Warehousing Professionals Within the framework of a project, data warehousing specialists are responsible for developing data management processes across a company. Furthermore, they construct software applications and computer programs for accomplishing data storage and management.

Data Science 98

Data Science Business Analyst Data Architect ETL Method 98

Cloudera

AUGUST 9, 2022

Customers, especially those in regulated industries with strict data protection and compliance requirements, routinely ask a straightforward question of our technical strategy experts: what should I do if a catastrophe hits my business and threatens to take out my data platform? The CDP Disaster Recovery Reference Architecture.

Data Lake 84

Data Lake Data Warehouse Architecture Data Ingestion 84

Cloudera

NOVEMBER 23, 2021

HBase is a column-oriented data storage architecture that is formed on top of HDFS to overcome its limitations. Although the HBase architecture is a NoSQL database, it eases the process of maintaining data by distributing it evenly across the cluster. Apache HBase.

Database 96

Database Non-relational Database NoSQL Government 96