Confluent

JUNE 13, 2019

You might think that data collection in astronomy consists of a lone astronomer pointing a telescope at a single object in a static sky. While that may be true in some cases (I collected the data for my Ph.D. thesis this way), the field of astronomy is rapidly changing into a data-intensive science with real-time needs.

Kafka 102

Kafka Bytes Data Pipeline Transportation 102

ProjectPro

JUNE 6, 2025

Apache Spark Streaming Use Cases Spark Streaming Architecture: Discretized Streams Spark Streaming Example in Java Spark Streaming vs. Structured Streaming Spark Streaming Structured Streaming What is Kafka Streaming? Kafka Stream vs. Spark Streaming What is Spark streaming? live logs, IoT device data, system telemetry data, etc.)

Architecture 45

Architecture Kafka Java Scala 45

ProjectPro

JUNE 6, 2025

These DStreams allow developers to cache data in memory, which may be particularly handy if the data from a DStream is utilized several times. The cache() function or the persist() method with proper persistence settings can be used to cache data. ’ A DataFrame is an immutable distributed columnar data collection.

Hadoop 68

Hadoop Metadata Java Datasets 68

ProjectPro

JUNE 6, 2025

Becoming a Big Data Engineer - The Next Steps Big Data Engineer - The Market Demand An organization’s data science capabilities require data warehousing and mining, modeling, data infrastructure, and metadata management. Most of these are performed by Data Engineers.

Big Data 40

Big Data Data Engineering Data Engineer Engineering 40

DoorDash Engineering

AUGUST 15, 2023

Here we will take a look at how we built BPFAgent, the process of building and maintaining its probes, and how various DoorDash teams have used the data collected. We also have an unmarshalling function to convert the raw bytes from the kernel into our structure. struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx); if (!sk)

Bytes 84

Bytes PostgreSQL Coding Database 84

ProjectPro

DECEMBER 28, 2021

Apache Spark Streaming Use Cases Spark Streaming Architecture: Discretized Streams Spark Streaming Example in Java Spark Streaming vs. Structured Streaming Spark Streaming Structured Streaming What is Kafka Streaming? Kafka Stream vs. Spark Streaming What is Spark streaming? live logs, IoT device data, system telemetry data, etc.)

Architecture 52

Architecture Kafka Java Scala 52

ProjectPro

JUNE 6, 2025

The World Economic Forum predicts that by 2025, 463 exabytes of data will be produced daily across the world. Exabytes are 10006 bytes, so to put it into perspective, 463 exabytes is the same as 212,765,957 DVDs. Expertise in creating scalable and efficient data processing architectures and also, monitor data processing systems.

Certification 40

Certification Data Engineering Data Engineer Engineering 40