Snowflake

NOVEMBER 11, 2024

Data and AI architecture matter “Before focusing on AI/ML use cases such as hyper personalization and fraud prevention, it is important that the data and data architecture are organized and structured in a way which meets the requirements and standards of the local regulators around the world.

Banking 98

Banking Finance Retail Food 98

Monte Carlo

OCTOBER 31, 2024

Key Differences Between AI Data Engineers and Traditional Data Engineers While traditional data engineers and AI data engineers have similar responsibilities, they ultimately differ in where they focus their efforts.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Snowflake

JANUARY 30, 2023

After four months of testing, du Toit and his team had moved one of its databases to Snowflake for bulk data processing. Ramp fetches and delivers data into S3 buckets as well as uses dbt to transform data at each stage. “Snowflake quickly beat the other vendors we were looking at in terms of price and features.”

Data Architecture 52

Data Architecture Architecture Management Datasets 52

phData: Data Engineering

NOVEMBER 8, 2024

Additionally, the optimized query execution and data pruning features reduce the compute cost associated with querying large datasets. Scaling data infrastructure while maintaining efficiency is one of the primary challenges of modern data architecture.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Edureka

APRIL 22, 2025

Its multi-cluster shared data architecture is one of its primary features. Additionally, Fabric has deep integrations with Power BI for visualization and Microsoft Purview for governance, resulting in a smooth experience for both business users and data professionals.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Striim

NOVEMBER 17, 2023

Striim serves as a real-time data integration platform that seamlessly and continuously moves data from diverse data sources to destinations such as cloud databases, messaging systems, and data warehouses, making it a vital component in modern data architectures.

Machine Learning 52

Machine Learning Data Process PostgreSQL Process 52

AltexSoft

JULY 18, 2023

It allows data scientists to analyze large datasets and interactively run jobs on them from the R shell. Big data processing. Despite these nuances, Spark’s high-speed processing capabilities make it an attractive choice for big data processing tasks. Here are some of the possible use cases.

Big Data 52

Big Data Data Process Process Hadoop 52

DataKitchen

NOVEMBER 5, 2024

The Race For Data Quality In A Medallion Architecture The Medallion architecture pattern is gaining traction among data teams. It is a layered approach to managing and transforming data. It sounds great, but how do you prove the data is correct at each layer? How do you ensure data quality in every layer

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Striim

MARCH 21, 2025

Before loading the data to Snowflake with sub-second latency, Striim allows users to perform in-line transformations, including denormalization, filtering, enrichment and masking, using a SQL-based language. In-flight data processing reduces the time needed for data preparation as it delivers the data in a consumable form.

Data Warehouse 52

Data Warehouse MongoDB MySQL Hadoop 52

Cloudera

FEBRUARY 15, 2024

Seeing the future in a modern data architecture The key to successfully navigating these challenges lies in the adoption of a modern data architecture.

Insurance 88

Insurance Banking Data Architecture Data Ingestion 88

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Data Engineering Weekly

DECEMBER 2, 2023

The Current State of the Data Architecture S3 intelligent tiered storage provides a fine balance between the cost and the duration of the data retention. However, the real-time insight on accessing the recent data remains a big challenge. The combination of stream processing + OLAP storage like Pinot. What is Next?

IT 85

IT BI AWS Kafka 85

Cloudera

OCTOBER 19, 2023

While navigating so many simultaneous data-dependent transformations, they must balance the need to level up their data management practices—accelerating the rate at which they ingest, manage, prepare, and analyze data—with that of governing this data.

Telecommunication 76

Telecommunication Data Architecture Government Architecture 76

Striim

SEPTEMBER 11, 2024

Data pipelines are the backbone of your business’s data architecture. Implementing a robust and scalable pipeline ensures you can effectively manage, analyze, and organize your growing data. Understanding the essential components of data pipelines is crucial for designing efficient and effective data architectures.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

phData: Data Engineering

FEBRUARY 21, 2025

On-prem data warehouses can provide lower latency solutions for critical applications that require high performance and low latency. Many companies may choose an on-prem data warehousing solution for quicker data processing to enable business decisions. Data integrations and pipelines can also impact latency.

Cloud 52

Cloud Data Warehouse Amazon Web Services Data Ingestion 52

Netflix Tech

NOVEMBER 14, 2023

In this context, managing the data, especially when it arrives late, can present a substantial challenge! In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Let’s dive in! To solve these problems, we came up with Psyberg!

Data Engineering 108

Data Engineering Data Engineer Engineering Metadata 108

Cloudera

OCTOBER 7, 2021

Those decentralization efforts appeared under different monikers through time, e.g., data marts versus data warehousing implementations (a popular architectural debate in the era of structured data) then enterprise-wide data lakes versus smaller, typically BU-Specific, “data ponds”.

Architecture 124

Architecture Metadata Kafka Government 124

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 95

Kafka Technology IT Government 95

Snowflake

JUNE 21, 2024

Data infrastructure should serve the current set of business needs and be able to scale and evolve with change. With Snowflake and Iceberg tables, customers have the ability to adapt to these changes and deploy their choice of data architecture, all while maintaining leading security, performance and simplicity.

Data Lake 124

Data Lake BI Business Intelligence Metadata 124

Cloudera

MARCH 23, 2022

In fact, we recently announced the integration with our cloud ecosystem bringing the benefits of Iceberg to enterprises as they make their journey to the public cloud, and as they adopt more converged architectures like the Lakehouse. Change data capture (CDC). 1: Multi-function analytics . Flexible and open file formats.

Metadata 92

Metadata Data Architecture Machine Learning BI 92

AltexSoft

FEBRUARY 11, 2023

This specialist works closely with people on both business and IT sides of a company to understand the current needs of the stakeholders and help them unlock the full potential of data. To get a better understanding of a data architect’s role, let’s clear up what data architecture is.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. Spark: The definitive guide: Big data processing made simple. O’Reilly Media, Inc.” [2]

Data Lake 98

Data Lake Data Warehouse Hadoop Architecture 98

Cloudera

NOVEMBER 18, 2020

The technological linchpin of its digital transformation has been its Enterprise Data Architecture & Governance platform. It hosts over 150 big data analytics sandboxes across the region with over 200 users utilizing the sandbox for data discovery.

Medical 123

Medical Banking Telecommunication Government 123

Knowledge Hut

MAY 3, 2024

A new breed of ‘Fast Data’ architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. Dean Wampler (Renowned author of many big data technology-related books) Dean Wampler makes an important point in one of his webinars. Dataflow 4.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Cloudera

JUNE 18, 2022

In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake. In a rush to own this term, many vendors have lost sight of the fact that the openness of a data architecture is what guarantees its durability and longevity.

Data Lake 94

Data Lake Data Warehouse BI SQL 94

Data Engineering Podcast

OCTOBER 14, 2019

We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council. Upcoming events include the combined events of the Data Architecture Summit and Graphorum, the Data Orchestration Summit, and Data Council in NYC.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Cloudera

MAY 8, 2023

Out of the box Cloudera Data platform (CDP) performs superbly but over time, if data architecture, data engineering, and DevOps best practices are not maintained, you can get stuck maintaining the wild, wild west. In this six-part series, we’re focused on improving the health of your environment.

Professional Services 77

Professional Services Data Architecture Data Engineering Data Engineer 77

Databand.ai

AUGUST 30, 2023

A DataOps architecture is the structural foundation that supports the implementation of DataOps principles within an organization. It encompasses the systems, tools, and processes that enable businesses to manage their data more efficiently and effectively. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Weekly

DECEMBER 10, 2023

From a data architecture point of view, this enables a lot of flexibility in integrating multiple systems. link] Netflix: Diving Deeper into Psyberg: Stateless vs Stateful Data Processing Netflix wrote a deep-dive article about Psyberg’s incremental data processing pipeline framework.

Data Engineering 77

Data Engineering Data Engineer Engineering Metadata 77

Snowflake

JULY 22, 2024

Determining an architecture and a scalable data model to integrate more source systems in the future. The benefits of migrating to Snowflake start with its multi-cluster shared data architecture, which enables scalability and high performance. Features such as auto-suspend and a pay-as-you-go model help you save costs.

Government 119

Government Data Cloud Hadoop 119

Data Engineering Weekly

SEPTEMBER 18, 2024

The Rise of Data Observability Data observability has become increasingly critical as companies seek greater visibility into their data processes. This growing demand has found a natural synergy with the rise of the data lake.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Data 98

DataKitchen

DECEMBER 30, 2020

Why we are watching them: DataKitchen aims to automate and coordinate people, tools and environments of an entire data analytic organization. The company also works to help organizations on applying Agile, DevOps and Lean principles to their data processes with the DataOps Cookbook and manifesto.

Manufacturing 96

Manufacturing Big Data Data Science Data Pipeline 96

Snowflake

AUGUST 1, 2024

Snowpark Container Services gives developers the ability to bring any containerized workload to their data that is already secure in Snowflake — ReactJS front-ends, open source large language models (LLMs), distributed data processing pipelines, you name it.

Deep Learning 125

Deep Learning Government AWS Architecture 125

Cloudera

NOVEMBER 29, 2021

eMAG , a Romania-based retailer seen as a pioneer in e-commerce, was struggling to manage the tremendously large amount of data coming in every second. The company needed a modern data architecture to manage the growing traffic effectively. . This creates long delays in data processing, which halts efficient functioning. .

Data Lake 77

Data Lake Telecommunication Retail Data 77

Striim

AUGUST 5, 2024

Discover how Alex’s journey from building racing motorcycles and tattoo machines as a child led him to revolutionize stream processing and cloud infrastructure. Explore the intricate challenges and groundbreaking innovations in data storage and streaming.

Kafka 52

Kafka Data Storage Architecture Data Architecture 52

Cloudyard

JULY 22, 2024

Read Time: 2 Minute, 57 Second Previously, data engineers used Kinesis Firehose to transfer data into blob storage (S3) and then load it into Snowflake using either Snowpipe or batch processing. This introduced latency in the data pipeline for near real-time data processing.

AWS 52

AWS Data Ingestion Data Architecture Architecture 52

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. Apache ORC (Optimized Row Columnar) : In 2013, ORC was developed for the Hadoop ecosystem to improve the efficiency of data storage and retrieval.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52