Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Cloudera

FEBRUARY 9, 2021

Today’s customers have a growing need for a faster end to end data ingestion to meet the expected speed of insights and overall business demand. This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability.

Data Warehouse 72

Data Warehouse Cloud Kafka Cloud Storage 72

RandomTrees

SEPTEMBER 6, 2020

The architecture is three layered: Database Storage: Snowflake has a mechanism to reorganize the data into its internal optimized, compressed and columnar format and stores this optimized data in cloud storage. The data objects are accessible only through SQL query operations run using Snowflake.

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

ProjectPro

FEBRUARY 6, 2023

Data engineers often use Google Cloud Pub/Sub to design asynchronous workflows, publish event notifications, and stream data from several processes or devices. This blog provides an overview of Google Cloud Pub/Sub that will help you understand the framework and its suitable use cases for your data engineering projects.

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

Cloudera

FEBRUARY 16, 2022

In the case of CDP Public Cloud, this includes virtual networking constructs and the data lake as provided by a combination of a Cloudera Shared Data Experience (SDX) and the underlying cloud storage. Each project consists of a declarative series of steps or operations that define the data science workflow.

Machine Learning 85

Machine Learning Algorithm Government Metadata 85

ProjectPro

AUGUST 24, 2021

Data professionals who work with raw data like data engineers, data analysts, machine learning scientists , and machine learning engineers also play a crucial role in any data science project. And, out of these professions, this blog will discuss the data engineering job role.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Ascend.io

JUNE 6, 2023

Understanding and controlling cloud costs is a fundamental part of how Ascend manages the cloud infrastructure of our dedicated deployment customers. These are customers where the entire Ascend software stack is installed in their cloud account. cents per gigabyte. cents per gigabyte.

Cloud 52

Cloud Data Pipeline Data Ingestion Cloud Storage 52

Netflix Tech

DECEMBER 14, 2018

In the previous blog posts in this series, we introduced the N etflix M edia D ata B ase ( NMDB ) and its salient “Media Document” data model. A fundamental requirement for any lasting data system is that it should scale along with the growth of the business applications it wishes to serve.

Media 97

Media Database Metadata Data Schemas 97

Rockset

MARCH 15, 2021

While there’s typically some amount of data engineering required here, there are ways to minimize it. For example, instead of denormalizing the data, you could use a query engine that supports joins. This will avoid unnecessary processing during data ingestion and reduce the storage bloat due to redundant data.

MongoDB 40

MongoDB Data Ingestion Analytics Application Kafka 40

Rockset

NOVEMBER 12, 2020

In most scenarios, MongoDB can be used as the primary data storage for write-only operations and as support for quick data ingestion. This blog post will examine the various tools that can be used to sync data between MongoDB and Elasticsearch.

MongoDB 40

MongoDB NoSQL Data Pipeline Data Storage 40

Knowledge Hut

APRIL 25, 2023

Key features of Amazon Redshift: Columnar storage for efficient data storage and retrieval Advanced compression techniques for reducing storage costs Automatic optimization of queries for faster performance Integration with AWS data lake services for easy data ingestion Scalability and elasticity to handle growing data volumes 2.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Knowledge Hut

JULY 6, 2023

IT Professionals looking to work in the cloud domain are expected to have a sound understanding of Azure tools as well as development and monitoring tools. This blog walks you through the top Azure Monitoring and Development that every SRE and DevOps engineer must know. However, there are costs associated with data ingestion.

Amazon Web Services 52

Amazon Web Services Data Lake Java SQL 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. However, you can also pull data from centralized data sources like data warehouses to transform data further and build ETL pipelines for training and evaluating AI agents.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 59

Data Ingestion Kafka Java Data Pipeline 59

Cloudera

OCTOBER 15, 2020

About this Blog. Data Discovery and Exploration (DDE) was recently released in tech preview in Cloudera Data Platform in public cloud. In this blog we will go through the process of indexing data from S3 into Solr in DDE with the help of NiFi in Data Flow. logs, twitter feeds, file appends etc).

AWS 118

AWS Data Cloud Accessibility 118

ProjectPro

MAY 31, 2021

Ace your big data interview by adding some unique and exciting Big Data projects to your portfolio. This blog lists over 20 big data projects you can work on to showcase your big data skills and gain hands-on experience in big data tools and technologies. How Big Data Works?

Big Data 52

Big Data Coding Project Hadoop 52

Data Engineering Weekly

JANUARY 24, 2025

The world of data management is undergoing a rapid transformation. The rise of cloud storage, coupled with the increasing demand for real-time analytics, has led to the emergence of the Data Lakehouse. This paradigm combines the flexibility of data lakes with the performance and reliability of data warehouses.

Architecture 77

Architecture Metadata Data Ingestion Data Lake 77

Edureka

APRIL 15, 2025

. “Microsoft Fabric Data Engineer Associate ” is the official title of the DP-700, which is intended to verify professionals’ proficiency in using Microsoft Fabric to create reliable data solutions. It validates expertise in data ingestion, transformation, security, and optimization within the Fabric ecosystem.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52