Rockset

MARCH 1, 2023

When you deconstruct the core database architecture, deep in the heart of it you will find a single component that is performing two distinct competing functions: real-time data ingestion and query serving. When data ingestion has a flash flood moment, your queries will slow down or time out making your application flaky.

Data Ingestion 144

Data Ingestion Database Architecture SQL 144

Snowflake

JUNE 4, 2024

Iceberg tables (now generally available), when combined with the capabilities of the Snowflake platform, allow you to build various open architectures, including a data lakehouse and data mesh. Parquet Direct (private preview) allows you to use Iceberg without rewriting or duplicating Parquet files — even as new Parquet files arrive.

Government 86

Government Data Ingestion Data PostgreSQL 86

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Effective Data Anomaly Monitoring (#2) Introduction Ensuring the accuracy and timeliness of data ingestion is a cornerstone for maintaining the integrity of data systems. This process is critical as it ensures data quality from the onset.

Data Ingestion 53

Data Ingestion Transportation High Quality Data Data Schemas 53

Snowflake

APRIL 18, 2024

This blog post explores how Snowflake can help with this challenge. Legacy SIEM cost factors to keep in mind Data ingestion: Traditional SIEMs often impose limits to data ingestion and data retention. Now there are a few ways to ingest data into Snowflake.

Data Lake 83

Data Lake Data Ingestion Bytes Cloud Computing 83

Striim

SEPTEMBER 11, 2024

Understanding the essential components of data pipelines is crucial for designing efficient and effective data architectures. Data Collection/Ingestion The next component in the data pipeline is the ingestion layer, which is responsible for collecting and bringing data into the pipeline.

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Snowflake

APRIL 9, 2024

However, that data must be ingested into our Snowflake instance before it can be used to measure engagement or help SDR managers coach their reps — and the existing ingestion process had some pain points when it came to data transformation and API calls.

BI 79

BI Data Ingestion Data Aggregated Data 79

Cloudera

FEBRUARY 15, 2024

Some of the key benefits of a modern data architecture for regulatory compliance include: Enhanced data governance and compliance: Modern data architecture incorporates data governance practices and security controls to ensure data privacy, regulatory compliance, and protection against unauthorized access or breaches.

Insurance 87

Insurance Banking Data Architecture Data Ingestion 87

DataKitchen

NOVEMBER 5, 2024

By systematically moving data through these layers, the Medallion architecture enhances the data structure in a data lakehouse environment. This architecture is valuable for organizations dealing with large volumes of diverse data sources, where maintaining accuracy and accessibility at every stage is a priority.

Architecture 79

Architecture Raw Data Pipeline-centric Data Ingestion 79

Striim

JUNE 6, 2024

Conversely, high latency can hinder your organization’s data integration and streaming efforts. As data-driven decision-making becomes increasingly vital, the importance of minimizing latency has never been clearer. Operational inefficiencies: Delayed data transmission results in operational processing disruptions.

Data Integration 52

Data Integration Data Ingestion Healthcare Data Pipeline 52

Data Engineering Weekly

SEPTEMBER 18, 2024

End-to-End Observability: The tool should provide full visibility across the entire pipeline, including data ingestion, transformation, and consumption. It should offer: - Data lineage tracking: Understand the flow and transformations of data through various systems.

Data Lake 98

Data Lake Data Pipeline Unstructured Data Datasets 98

Rockset

JANUARY 30, 2024

With this architecture, users can separate ingestion compute from query compute, all while accessing the same real-time data. Microbatching : An option to microbatch ingestion based on the latency requirements of the use case. This is not a hands-free operation and also involves the transfer of data across nodes.

Data Ingestion 64

Data Ingestion Utilities Architecture SQL 64

Cloudera

FEBRUARY 9, 2021

Cloudera Operational Database is now available in three different form-factors in Cloudera Data Platform (CDP). . If you are new to Cloudera Operational Database, see this blog post. In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database.

Database 103

Database Java Data Ingestion SQL 103

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. Accelerated Data Analytics DataOps tools help automate and streamline various data processes, leading to faster and more efficient data analytics.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Edureka

AUGUST 23, 2024

The AWS training will prepare you to become a master of the cloud, storing, processing, and developing applications for the cloud data. Amazon AWS Kinesis makes it possible to process and analyze data from multiple sources in real-time. What can I do with Kinesis Data Streams? Table of Content What is Amazon Kinesis?

AWS 52

AWS Kafka Amazon Web Services Medical 52

Databand.ai

AUGUST 30, 2023

The core philosophy of DataOps is to treat data as a valuable asset that must be managed and processed efficiently. It emphasizes the importance of collaboration between different teams, such as data engineers, data scientists, and business analysts, to ensure that everyone has access to the right data at the right time.

Data Governance 52

Data Governance Data Pipeline Government Business Analyst 52

Cloudyard

JULY 16, 2024

Read Time: 3 Minute, 11 Second This blog post showcases a real-time data pipeline built in Snowflake that leverages Slowly Changing Dimensions (SCD 2) and Finalizer Tasks to ensure your customer data is always fresh, accurate, and reflects historical changes.

Data Ingestion 52

Data Ingestion Data Pipeline Metadata Utilities 52

Edureka

FEBRUARY 9, 2023

As a certified Azure Data Engineer, you have the skills and expertise to design, implement and manage complex data storage and processing solutions on the Azure cloud platform. As the demand for data engineers grows, having a well-written resume that stands out from the crowd is critical.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

phData: Data Engineering

MARCH 8, 2023

Fivetran, a cloud-based automated data integration platform, has emerged as a leading choice among businesses looking for an easy and cost-effective way to unify their data from various sources. With over 160 data connectors available, Fivetran makes it easy to move data out of, into, and across any cloud data platform in the market.

IT 96

IT Data Warehouse Data Ingestion Data Integration 96

Cloudera

MAY 3, 2024

The workflow—from data ingestion and model training to model deployment—is meticulously defined within a YAML configuration file. HuggingFace has built an unmatched community of the best and brightest data scientists, and Spaces are where this community shares its best projects.

Data Security 69

Data Security Machine Learning Data Ingestion Professional Services 69

Snowflake

SEPTEMBER 13, 2023

Snowpark External Access – public preview External Access is in public preview on AWS regions. Users can now easily connect to external network locations, including external LLMs, from their Snowpark code while maintaining high security and governance over their data. Snowpark Python Updates Snowpark support for Python 3.9

Python 81

Python SQL Data Pipeline Data Ingestion 81

phData: Data Engineering

NOVEMBER 8, 2024

Open Table Format (OTF) architecture now provides a solution for efficient data storage, management, and processing while ensuring compatibility across different platforms. In this blog, we will discuss: What is the Open Table format (OTF)? They also support ACID transactions, ensuring data integrity and stored data reliability.

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Cloudera

OCTOBER 15, 2020

About this Blog. Data Discovery and Exploration (DDE) was recently released in tech preview in Cloudera Data Platform in public cloud. In this blog we will go through the process of indexing data from S3 into Solr in DDE with the help of NiFi in Data Flow. Spark as the ingest pipeline tool for Search (i.e.

AWS 116

AWS Data Cloud Cloud Storage 116

Cloudera

APRIL 9, 2021

This is part 4 in this blog series. This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The second blog dealt with creating and managing Data Enrichment pipelines.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). In this blog post, we’ll see how you can use other CDP services with COD. Integrated across the Enterprise Data Lifecycle . Cloudera Shared Data Experience (SDX) . Conclusion.

Database 81

Database Machine Learning Kafka Aggregated Data 81

Cloudera

SEPTEMBER 9, 2022

The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested.

Architecture 101

Architecture Metadata Machine Learning Unstructured Data 101

Cloudera

DECEMBER 14, 2021

During the economic upheaval brought about by the COVID pandemic, the government of Australia was working with the Commonwealth Bank of Australia (CBA) to access real-time data about everyday financial transactions to understand the economic and social impacts of the crisis. Commonwealth Bank of Australia.

Transportation 110

Transportation Telecommunication Banking Data Lake 110

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. You can read part 1, here: Digital Transformation is a Data Journey From Edge to Insight. The first blog introduced a mock connected vehicle manufacturing company, The Electric Car Company (ECC), to illustrate the manufacturing data path through the data lifecycle.

Data Pipeline 117

Data Pipeline Building Manufacturing Data Warehouse 117

Cloudera

DECEMBER 20, 2021

The right set of tools helps businesses utilize data to drive insights and value. But balancing a strong layer of security and governance with easy access to data for all users is no easy task. You can become a data hero too. Retrofitting existing solutions to ever-changing policy and security demands is one option.

Government 89

Government Data Security Banking Metadata 89

Cloudera

JULY 9, 2021

If you’ve followed Cloudera for a while, you know we’ve long been singing the praises—or harping on the importance, depending on perspective—of a solid, standalone enterprise data strategy. The ways data strategies are implemented, the resulting outcomes and the lessons learned along the way provide important guardrails.

Government 113

Government Building Cloud Data Ingestion 113

Cloudyard

APRIL 4, 2024

However, the volume of daily transaction data poses challenges in effectively segmenting customers and optimizing engagement. This blog post explores how Snowpark, a powerful tool for data processing within Snowflake, can be used to perform RFM segmentation and unlock actionable customer insights.

Retail 40

Retail Data Ingestion Metadata Database 40

Cloudera

DECEMBER 15, 2022

Since we announced the general availability of Apache Iceberg in Cloudera Data Platform (CDP), Cloudera customers, such as Teranet , have built open lakehouses to future-proof their data platforms for all their analytical workloads. Cloudera partners are also benefiting from Apache Iceberg in CDP. Amazingly fast table migration.

Cloud 78

Cloud Metadata Google Cloud Data Warehouse 78

Snowflake

AUGUST 19, 2024

In this blog, we will cover some of the most recently launched improvements for the Snowflake platform. Intelligent workload optimization features allow customers to improve query performance, access insights and optimize storage and computing.

Data Ingestion 52

Data Ingestion BI Structured Data Engineering 52

Cloudera

APRIL 26, 2023

Data is core to decision making today and organizations often turn to the cloud to build modern data apps for faster access to valuable insights. Can you achieve similar outcomes with your on-premises data platform? The post Building Cloud Native Data Apps on Premises appeared first on Cloudera Blog.

Cloud 63

Cloud Building Utilities Architecture 63

Pinterest Engineering

AUGUST 18, 2022

The HBase Ecosystem, though having various advantages like strong consistency at row level in high volume requests, flexible schema, low latency access to data, Hadoop integration, etc. In this blog post, we will first learn the various approaches considered for data migration with their trade offs.

Data Ingestion 52

Data Ingestion Hadoop Database Kafka 52

ProjectPro

FEBRUARY 6, 2023

Data engineers often use Google Cloud Pub/Sub to design asynchronous workflows, publish event notifications, and stream data from several processes or devices. This blog provides an overview of Google Cloud Pub/Sub that will help you understand the framework and its suitable use cases for your data engineering projects.

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

Cloudera

MAY 3, 2024

The workflow—from data ingestion and model training to model deployment—is meticulously defined within a YAML configuration file. HuggingFace has built an unmatched community of the best and brightest data scientists, and Spaces are where this community shares its best projects.

Project 52

Project Machine Learning Data Ingestion Professional Services 52