Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. You need to think about the whole model lifecycle.

Machine Learning 107

Machine Learning Python Kafka Java 107

Cloudera

FEBRUARY 9, 2021

Today’s customers have a growing need for a faster end to end data ingestion to meet the expected speed of insights and overall business demand. This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability.

Data Warehouse 72

Data Warehouse Cloud Kafka Cloud Storage 72

Towards Data Science

MARCH 5, 2024

BigQuery separates storage and compute with Google’s Jupiter network in-between to utilize 1 Petabit/sec of total bisection bandwidth. The storage system is using Capacitor, a proprietary columnar storage format by Google for semi-structured data and the file system underneath is Colossus, the distributed file system by Google.

Bytes 97

Bytes Google Cloud Cloud Storage Utilities 97

ProjectPro

FEBRUARY 6, 2023

With over 10 million active subscriptions, 50 million active topics, and a trillion messages processed per day, Google Cloud Pub/Sub makes it easy to build and manage complex event-driven systems. Google Cloud Pub/Sub is a messaging service that allows apps and services to exchange event data.

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

DareData

JANUARY 30, 2023

Our goal is to help data scientists better manage their models deployments or work more effectively with their data engineering counterparts, ensuring their models are deployed and maintained in a robust and reliable way. DigDag: An open-source orchestrator for data engineering workflows.

Data Engineering 52

Data Engineering Data Engineer NoSQL Engineering 52

Towards Data Science

OCTOBER 9, 2024

Get started with Airbyte and Cloud Storage Coding the connectors yourself? Think very carefully Creating and maintaining a data platform is a hard challenge. Data connectors are an essential part of such a platform. Of course, how else are we going to get the data? So, do what is best for your application.

Coding 75

Coding Cloud Storage Data Data Ingestion 75

Ascend.io

FEBRUARY 23, 2024

If you are in a private data center, this might be the reason you finally open up that cloud account. If your core data systems are still running in a private data center or pushed to VMs in the cloud, you have some work to do. Robust Data Ingestion AI systems thrive on diverse data sources.

Cloud Storage 52

Cloud Storage Data Ingestion Machine Learning Algorithm 52

phData: Data Engineering

AUGUST 4, 2023

Data storage is a vital aspect of any Snowflake Data Cloud database. Within Snowflake, data can either be stored locally or accessed from other cloud storage systems. What are the Different Storage Layers Available in Snowflake?

Cloud Storage 52

Cloud Storage Google Cloud Amazon Web Services Data Storage 52

Rockset

MARCH 5, 2021

How Snowflake handles space-time tradeoff When data is loaded into Snowflake, it reorganizes that data into its compressed, columnar format and stores it in cloud storage - this means it is highly optimized for space which directly translates to minimizing your storage footprint.

Cloud Storage 52

Cloud Storage Data Ingestion Data Warehouse Computer Science 52

Confluent

OCTOBER 16, 2019

Additional data is available over REST as well as static reference data published on web pages. As with any system out there, the data often needs processing before it can be used. As with any real system, the data has “character.” Instead of using system time , we want to work with event time.

Kafka 19

Kafka Building Data Coding 19

AltexSoft

MAY 12, 2023

Generated by various systems or applications, log files usually contain unstructured text data that can provide insights into system performance, security, and user behavior. Sensor data. A fixed schema means the structure and organization of the data are predetermined and consistent. Scalability.

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

Rockset

MARCH 21, 2019

When we started Rockset, we envisioned building a powerful cloud data management system that was really easy to use. Making the data stack simpler is fundamental to making data usable by developers and data scientists. Another key aspect of Rockset that makes it simple to use is its serverless nature.

SQL 52

SQL Data Management Management Engineering 52

ProjectPro

AUGUST 24, 2021

Data Engineering Project for Beginners If you are a newbie in data engineering and are interested in exploring real-world data engineering projects, check out the list of data engineering project examples below. This big data project discusses IoT architecture with a sample use case.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Knowledge Hut

JUNE 26, 2023

Finnhub API with Kafka for Real-Time Financial Market Data Pipeline Project Overview: The goal of this project is to construct a streaming data pipeline by making use of the real-time financial market data API provided by Finnhub. In addition to this, they make sure that the data is always readily accessible to consumers.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Netflix Tech

DECEMBER 14, 2018

In the previous blog posts in this series, we introduced the N etflix M edia D ata B ase ( NMDB ) and its salient “Media Document” data model. In this post we will provide details of the NMDB system architecture beginning with the system requirements?—?these key value stores generally allow storing any data under a key).

Media 97

Media Database Metadata Data Schemas 97

Ascend.io

JUNE 6, 2023

To gain a concrete understanding and provide tangible insights for data pipeline optimization , we’ve monitored the performance of one of our production pipeline networks — an established system that handles significant data volumes and undergoes updates approximately every 30 minutes. cents per gigabyte.

Cloud 52

Cloud Data Pipeline Data Ingestion Cloud Storage 52

Rockset

MARCH 15, 2021

When working with a real-time analytics system you need your database to meet very specific requirements. This includes making the data available for query as soon as it is ingested, creating proper indexes on the data so that the query latency is very low, and much more. Rockset takes a different approach here, too.

MongoDB 40

MongoDB Data Ingestion Analytics Application Kafka 40

Rockset

MARCH 1, 2023

Developers can spin up or down virtual instances based on the performance requirements of their streaming ingest or query workloads. In addition, Rockset provides fast data access through the use of more performant hot storage, while cloud storage is used for durability.

Architecture 59

Architecture AWS SQL Cloud Storage 59

AltexSoft

MARCH 30, 2023

Databricks architecture Databricks provides an ecosystem of tools and services covering the entire analytics process — from data ingestion to training and deploying machine learning models. Besides that, it’s fully compatible with various data ingestion and ETL tools. Let’s see what exactly Databricks has to offer.

Scala 64

Scala Data Lake BI Machine Learning 64

Knowledge Hut

SEPTEMBER 26, 2023

Born out of the minds behind Apache Spark, an open-source distributed computing framework, Databricks is designed to simplify and accelerate data processing, data engineering, machine learning, and collaborative analytics tasks. This flexibility allows organizations to ingest data from virtually anywhere.

Data Lake 52

Data Lake Database-centric Pipeline-centric Machine Learning 52

Monte Carlo

FEBRUARY 15, 2023

This article will define in simple terms what a data warehouse is, how it’s different from a database, fundamentals of how they work, and an overview of today’s most popular data warehouses. What is a data warehouse? An ETL tool or API-based batch processing/streaming is used to pump all of this data into a data warehouse.

Data Warehouse 52

Data Warehouse Unstructured Data AWS Business Intelligence 52

Monte Carlo

AUGUST 19, 2023

We’ll cover: What is a data platform? Recently, there’s been a lot of discussion around whether to go with open source or closed source solutions (the dialogue between Snowflake and Databricks’ marketing teams really brings this to light) when it comes to building your data platform.

Building 52

Building BI Data Lake Data Governance 52

Knowledge Hut

APRIL 25, 2023

Data processing: Data engineers should know data processing frameworks like Apache Spark, Hadoop, or Kafka, which help process and analyze data at scale. Data engineering tools can help data engineers streamline many of these tasks, allowing them to be more productive and effective in their work.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Monte Carlo

OCTOBER 5, 2023

Data consistency means your data should not contradict itself or other sources within the organization. Example: Product IDs should always be alphanumeric and maintain the same number of characters across all systems. Example: Social media metrics (e.g., likes, shares) should be refreshed at least once every 12 hours.

Government 52

Government Data Governance Data Cloud Storage 52

Knowledge Hut

JULY 6, 2023

Besides, it offers excellent managing and monitoring capabilities to help system admins and analysts increase productivity. Features The centralized data store integrates data from every system layer. Above all, it has built-in mechanisms to alert you whenever your system has a performance issue or security breach.

Amazon Web Services 52

Amazon Web Services Data Lake Java SQL 52

ProjectPro

DECEMBER 7, 2021

Data Pipeline Tools AWS Data Pipeline Azure Data Pipeline Airflow Data Pipeline Learn to Create a Data Pipeline FAQs on Data Pipeline What is a Data Pipeline? An ETL pipeline is a series of procedures that comprises extracting and transforming data from a data source.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Rockset

NOVEMBER 12, 2020

Elasticsearch is one tool to which reads can be offloaded, and, because both MongoDB and Elasticsearch are NoSQL in nature and offer similar document structure and data types, Elasticsearch can be a popular choice for this purpose. This blog post will examine the various tools that can be used to sync data between MongoDB and Elasticsearch.

MongoDB 40

MongoDB NoSQL Data Pipeline Data Storage 40

Knowledge Hut

AUGUST 18, 2023

Google Cloud Associate Cloud Engineer Certification (a) Certification Overview This Google platform certification is for individuals who have hands-on experience with Google Cloud & want to showcase their expertise in cloud technology. in the Google Cloud environment. (b)

Google Cloud 52

Google Cloud Certification Cloud Cloud Computing 52

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 59

Data Ingestion Kafka Java Data Pipeline 59

Monte Carlo

NOVEMBER 22, 2024

At the front end, you’ve got your data ingestion layer —the workhorse that pulls in data from everywhere it lives. Think of your data lake as a vast reservoir where you store raw data in its original form—great for when you’re not quite sure how you’ll use it yet.

Data Engineering 52

Data Engineering Data Engineer Building Engineering 52

AltexSoft

JULY 29, 2022

A Hadoop cluster is a group of computers called nodes that act as a single centralized system working on the same task. a client or edge node serves as a gateway between a Hadoop cluster and outer systems and applications. It loads data and grabs the results of the processing staying outside the master-slave hierarchy.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

MAY 31, 2021

Data Description: You will use the Covid-19 dataset(COVID-19 Cases.csv) from data.world , for this project, which contains a few of the following attributes: people_positive_cases_count county_name case_type data_source Language Used: Python 3.7 Machines and humans are both sources of structured data.

Big Data 52

Big Data Coding Project Hadoop 52

Data Engineering Weekly

DECEMBER 18, 2024

Inspired by the human brain, Neuromorphic chips promise unparalleled energy efficiency and the ability to process unstructured data locally on devices. The advancement in computing will expand AI’s role in autonomous systems and robotics. Tools like lakebyte.ai are the beginning of such a revolution.

Data Engineering 113

Data Engineering Data Engineer Engineering Data Lake 113

Data Engineering Weekly

JANUARY 24, 2025

The world of data management is undergoing a rapid transformation. The rise of cloud storage, coupled with the increasing demand for real-time analytics, has led to the emergence of the Data Lakehouse. This paradigm combines the flexibility of data lakes with the performance and reliability of data warehouses.

Architecture 76

Architecture Metadata Data Ingestion Data Lake 76

Edureka

APRIL 15, 2025

Officially titled “Implementing Data Engineering Solutions Using Microsoft Fabric” , this assessment evaluates a candidate’s ability to design and implement data engineering solutions using Microsoft Fabric. Data Factory : Automate workflows and manage data movement across multiple sources.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52