Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. show() The history object is a Spark Data Frame. delta_table.history().select("version",

Data Lake 76

Data Lake Data Warehouse Architecture Hadoop 76

Rockset

APRIL 26, 2023

We’ve noticed many common patterns across streaming data architectures and we’ll be sharing a blueprint for three of the most popular: anomaly detection, IoT, and recommendations. This will simplify your architecture considerably, deliver low-latency recommendation results, and enable metadata filtering.

Architecture 52

Architecture Transportation Data Lake Insurance 52

Rockset

AUGUST 2, 2023

The steps Windward takes to create proprietary data and AI insights As Windward operated in a batch-based data stack, they stored raw data in S3. They used MongoDB as their metadata store to capture vessel and company data.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Central to this transformation are two shifts.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

AltexSoft

MARCH 2, 2023

Data modeling involves creating a conceptual representation of data objects and their relationships to each other, as well as the rules governing those relationships. To design an effective data governance program, it’s crucial to choose an operational model that fits your business size and structure.

Data Governance 52

Data Governance Government Programming Healthcare 52

ProjectPro

JANUARY 31, 2023

Data Variety Hadoop stores structured, semi-structured and unstructured data. RDBMS stores structured data. Data storage Hadoop stores large data sets. RDBMS stores the average amount of data. Works with only structured data. Hardware Hadoop uses commodity hardware.

Big Data 52

Big Data Hadoop Relational Database AWS 52

ProjectPro

JANUARY 31, 2022

Snowflake in Action at Western Union Snowflake's multi-cluster shared data architecture expanded instantaneously to serve Western Union's data, users, and workloads without causing resource conflict. Snowflake saves and manages data on the cloud using a shared-disk approach, making data management simple.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

AltexSoft

JULY 18, 2023

Spark SQL brings native support for SQL to Spark and streamlines the process of querying semistructured and structured data. When transformations are applied to RDDs, Spark records the metadata to build up a DAG, which reflects the sequence of computations performed during the execution of the Spark job.

Big Data 52

Big Data Data Process Process Hadoop 52

Rockset

JANUARY 28, 2020

Aside from video data from each camera-equipped store, Standard deals with other data sets such as transactional data, store inventory data that arrive in different formats from different retailers, and metadata derived from the extensive video captured by their cameras.

Retail 40

Retail Google Cloud Raw Data Data Lake 40

Monte Carlo

JUNE 14, 2023

Data observability platforms reduce your data downtime up to 80% and make your data engineers 30% more time efficient by replacing static, cumbersome data testing with machine learning models that can help detect, resolve, and prevent data issues. Let the data drive the data pipeline architecture.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

AltexSoft

JULY 19, 2022

What data mesh is and is not. What data mesh IS. Data mesh is a set of principles for designing a modern distributed data architecture that focuses on business domains, not the technology used, and treats data as a product. For example, your organization has an HR platform that produces employee data.

Architecture 52

Architecture Data Lake Medical Datasets 52

ProjectPro

OCTOBER 20, 2021

Data Integration at Scale Most data architectures rely on a single source of truth. Having multiple data integration routes helps optimize the operational as well as analytical use of data. A feature store is a modern, elegant solution to leverage data prep work from previous runs or other teams as well.

Machine Learning 52

Machine Learning Algorithm Data Science Government 52

AltexSoft

MAY 12, 2023

What is unstructured data? Definition and examples Unstructured data , in its simplest form, refers to any data that does not have a pre-defined structure or organization. It can come in different forms, such as text documents, emails, images, videos, social media posts, sensor data, etc.

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

Cloudera

OCTOBER 7, 2021

Those decentralization efforts appeared under different monikers through time, e.g., data marts versus data warehousing implementations (a popular architectural debate in the era of structured data) then enterprise-wide data lakes versus smaller, typically BU-Specific, “data ponds”.

Architecture 124

Architecture Metadata Kafka Government 124

phData: Data Engineering

JULY 18, 2023

Demands on the cloud data warehouse are also evolving to require it to become more of an all-in-one platform for an organization’s analytics needs. Enter Snowflake The Snowflake Data Cloud is one of the most popular and powerful CDW providers.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Machine Learning 64