Cloudera

JUNE 28, 2022

In a previous blog of this series, Turning Streams Into Data Products , we talked about the increased need for reducing the latency between data generation/ingestion and producing analytical results and insights from this data. This blog will be published in two parts.

Process 86

Process Kafka SQL Machine Learning 86

Knowledge Hut

APRIL 25, 2023

An end-to-end Data Science pipeline starts from business discussion to delivering the product to the customers. One of the key components of this pipeline is Data ingestion. It helps in integrating data from multiple sources such as IoT, SaaS, on-premises, etc., What is Data Ingestion?

Data Ingestion 52

Data Ingestion Lambda Architecture Raw Data Data Science 52

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Rockset

MAY 3, 2023

To find out, we decided to test the streaming ingestion performance of Rockset’s next generation cloud architecture and compare it to open-source search engine Elasticsearch , a popular sink for Apache Kafka. For this benchmark, we evaluated Rockset and Elasticsearch ingestion performance on throughput and data latency.

Data Ingestion 59

Data Ingestion Kafka Database Architecture 59

Snowflake

JUNE 13, 2024

But at Snowflake, we’re committed to making the first step the easiest — with seamless, cost-effective data ingestion to help bring your workloads into the AI Data Cloud with ease. Like any first step, data ingestion is a critical foundational block. Ingestion with Snowflake should feel like a breeze.

Data Ingestion 91

Data Ingestion MySQL PostgreSQL Data Pipeline 91

Cloudera

MAY 19, 2021

In the previous blog post in this series, we walked through the steps for leveraging Deep Learning in your Cloudera Machine Learning (CML) projects. The script will go through loading RAPIDs libraries then leveraging them to load and processing a datafile. Data Ingestion. The raw data is in a series of CSV files.

Machine Learning 137

Machine Learning Datasets Data Science Raw Data 137

Cloudera

JANUARY 20, 2021

The missing chapter is not about point solutions or the maturity journey of use cases, the missing chapter is about the data, it’s always been about the data, and most importantly the journey data weaves from edge to artificial intelligence insight. . Data Collection Challenge.

Manufacturing 111

Manufacturing Data Warehouse Kafka Retail 111

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. You can read part 1, here: Digital Transformation is a Data Journey From Edge to Insight. The first blog introduced a mock connected vehicle manufacturing company, The Electric Car Company (ECC), to illustrate the manufacturing data path through the data lifecycle.

Data Pipeline 119

Data Pipeline Building Manufacturing Data Warehouse 119

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Data Engineering Weekly

JULY 7, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. Hudi seems to be a de facto choice for CDC data lake features. Notion migrated the insert heavy workload from Snowflake to Hudi.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Rockset

OCTOBER 11, 2022

As Rockset is purpose-built for real-time analytics, it has also been designed for field-level mutability , decreasing the CPU required to process inserts, updates and deletes. Logstash is an event processing pipeline that ingests and transforms data before sending it to Elasticsearch.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Cloudera

DECEMBER 14, 2021

So to improve the speed of data analysis, the IRS worked with the combined technology integrating Cloudera Data Platform (CDP) and NVIDIA’s RAPIDS Accelerator for Apache Spark 3.0. However, the CBA is a huge institution with 15 million customers and 700M daily transactions — managing the growing influx of data was challenging. .

Transportation 112

Transportation Telecommunication Banking Data Lake 112

Cloudera

FEBRUARY 8, 2021

Model accuracy is enabled by more accurate data collection and more accurate labeling and annotation, while the data reduction was achieved with a relevant selection of data for training and the ability to process and encode connected vehicle sensor data. . This author is passionate about industry 4.0,

Manufacturing 109

Manufacturing Machine Learning Data Ingestion Electronics 109

Cloudera

AUGUST 4, 2021

Using a scalable data management and analytics platform built on Cloudera Enterprise, Sikorsky can process and store data in a reliable way, and analyze full data sets across entire fleets. images, video, text, spectral data) or other input such as thermographic or acoustic signals. .

Unstructured Data 98

Unstructured Data Data Ingestion Government Machine Learning 98

Cloudera

FEBRUARY 9, 2021

Cloudera Operational Database is now available in three different form-factors in Cloudera Data Platform (CDP). . If you are new to Cloudera Operational Database, see this blog post. In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database.

Database 104

Database Java SQL Data Ingestion 104

Cloudera

OCTOBER 15, 2020

About this Blog. Data Discovery and Exploration (DDE) was recently released in tech preview in Cloudera Data Platform in public cloud. In this blog we will go through the process of indexing data from S3 into Solr in DDE with the help of NiFi in Data Flow. logs, twitter feeds, file appends etc).

AWS 118

AWS Data Cloud Accessibility 118

Cloudera

APRIL 20, 2021

Streaming Analytics is a type of data analysis that processes data streams for real-time analytics. It continuously processes data from multiple streams and performs simple calculations to complex event processing for delivering sophisticated use cases. What is Streaming Analytics?

Kafka 96

Kafka Hospitality Retail Data Ingestion 96

Cloudera

NOVEMBER 1, 2023

This AMP is built on the foundation of one of our previous AMP s, with the additional enhancement of enabling customers to create a knowledge base from data on their own website using Cloudera DataFlow (CDF) and then augment questions to the chatbot from that same knowledge base in Pinecone.

Machine Learning 111

Machine Learning Data Ingestion Database Architecture 111

Cloudera

NOVEMBER 17, 2020

Once you have deployed the template and all the CML artifacts that go with it, you can unpick and work it backward to map the process to your own data in your own environment. . The data and the techniques presented in this prototype are still applicable as creating a PCA feature store is often part of the machine learning process. .

Deep Learning 116

Deep Learning Machine Learning Raw Data Data Ingestion 116

Cloudera

AUGUST 13, 2021

Whether it is consuming log files, sensor metrics, and other unstructured data, most enterprises manage and deliver data to the data lake and leverage various applications like ETL tools, search engines, and databases for analysis. The ability to have the right data can determine an enterprise’s ability to succeed or fail.

Data Pipeline 111

Data Pipeline Data Lake ETL Tools Unstructured Data 111

Cloudera

AUGUST 11, 2022

And while operations in the cyber-domain are more likely to make the evening news, there are a vast array of critical use cases that support the military’s need for a data architecture that collects, processes, and delivers any type of data, anywhere. . edge processing. military installations spread across the globe.

Transportation 95

Transportation Data Ingestion Architecture Data 95

Pinterest Engineering

NOVEMBER 7, 2023

Jeff Xiang | Software Engineer, Logging Platform Vahid Hashemian | Software Engineer, Logging Platform Jesus Zuniga | Software Engineer, Logging Platform At Pinterest, data is ingested and transported at petabyte scale every day, bringing inspiration for our users to create a life they love.

Kafka 111

Kafka Java Software Engineer Software Engineering 111

Christophe Blefari

JANUARY 20, 2024

He wrote some years ago 3 articles defining data engineering field. Some concepts When doing data engineering you can touch a lot of different concepts. batch — Batch processing is at the core of data engineering. One of the major task is to move data from a source storage to a destination storage.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

phData: Data Engineering

MARCH 8, 2023

Fivetran, a cloud-based automated data integration platform, has emerged as a leading choice among businesses looking for an easy and cost-effective way to unify their data from various sources. It allows organizations to easily connect their disparate data sources without having to manage any infrastructure. Why Use Fivetran?

IT 96

IT Data Warehouse Data Ingestion Data Integration 96

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data.

Kafka 105

Kafka Data Ingestion Datasets Architecture 105

Cloudera

JUNE 7, 2021

CDF streamlines the process of collecting, curating and analyzing real-time streaming data with its integrated set of components. It calls out that Cloudera DataFlow “ includes streaming flow and streaming data processing unified with Cloudera Data Platform ”.

Kafka 104

Kafka Data Ingestion Cloud Architecture 104

Cloudera

AUGUST 26, 2020

In Part II of our Q&A, Dinesh will be looking at how businesses can leverage technology like Apache Flink and Apache NiFi to promote low latency processing of high-volume, high-velocity data. Hello Dinesh, thank you for joining us for Part II of our Q&A on streaming data.

Banking 85

Banking Data Ingestion Kafka Data Lake 85

DataKitchen

MAY 10, 2024

Harnessing Data Observability Across Five Key Use Cases The ability to monitor, validate, and ensure data accuracy across its lifecycle is not just a luxury—it’s a necessity. Data Evaluation Before new data sets are introduced into production environments, they must be thoroughly evaluated and cleaned.

Data Ingestion 60

Data Ingestion Datasets Data Coding 60

Christophe Blefari

MARCH 4, 2023

I'll try to think about it in the following weeks to understand where I go for the third year of the newsletter and the blog. The article has been written as something you can add in your own internal dbt onboarding process for every newcomer. So thank you for that. Stay tuned and let's jump to the content.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Snowflake

APRIL 18, 2024

This blog post explores how Snowflake can help with this challenge. Legacy SIEM cost factors to keep in mind Data ingestion: Traditional SIEMs often impose limits to data ingestion and data retention. These rules need computing power to analyze data and spot attacks.

Data Lake 83

Data Lake Data Ingestion Bytes Cloud Computing 83

Cloudera

JULY 9, 2021

This is to be expected, given the challenges that come with classified data and operations, legacy data of uncertain location and provenance, and thorny application rationalization processes that often uncover more, unexpected data problems, among other hurdles.

Government 114

Government Building Cloud Data Ingestion 114

Cloudera

DECEMBER 4, 2024

For more than a decade, Cloudera has been an ardent supporter and committee member of Apache NiFi, long recognizing its power and versatility for data ingestion, transformation, and delivery. empowers data engineers to build and deploy data pipelines faster, accelerating time-to-value for the business.

Data Pipeline 94

Data Pipeline Data Ingestion Data Preparation Architecture 94

Snowflake

OCTOBER 16, 2024

For organizations who are considering moving from a legacy data warehouse to Snowflake, are looking to learn more about how the AI Data Cloud can support legacy Hadoop use cases, or are struggling with a cloud data warehouse that just isn’t scaling anymore, it often helps to see how others have done it.

Digital Media 82

Digital Media Media Data Lake Data Warehouse 82

Striim

SEPTEMBER 11, 2024

With data volumes and sources rapidly increasing, optimizing how you collect, transform, and extract data is more crucial to stay competitive. That’s where real-time data, and stream processing can help. We’ll answer the question, “What are data pipelines?” Table of Contents What are Data Pipelines?

Data Pipeline 98

Data Pipeline Designing Data Lake Data Warehouse 98

Cloudera

FEBRUARY 15, 2024

Data integration and ingestion: With robust data integration capabilities, a modern data architecture makes real-time data ingestion from various sources—including structured, unstructured, and streaming data, as well as external data feeds—a reality.

Insurance 89

Insurance Banking Data Architecture Data Ingestion 89

Cloudera

DECEMBER 9, 2024

In this case, the Logistics AI assistant accesses data on truck maintenance and shipment timelines, enhancing decision-making for dispatchers and optimizing fleet schedules: RAG Architecture : User prompts are supplemented with additional context from knowledgebase and external lookups.

Professional Services 75

Professional Services Data Ingestion Manufacturing Retail 75

Cloudera

NOVEMBER 19, 2020

Analyzing historical data is an important strategy for anomaly detection. The modeling process begins with data collection. Here, Cloudera Data Flow is leveraged to build a streaming pipeline which enables the collection, movement, curation, and augmentation of raw data feeds.

Government 120

Government Machine Learning Algorithm Raw Data 120