Cloudera

JUNE 29, 2023

However, because most institutions lack a modern data architecture , they struggle to manage, integrate and analyze financial data at pace. Incorporate data from novel sources — social media feeds, alternative credit histories (utility and rental payments), geo-spatial systems, and IoT streams — into liquidity risk models.

Data Architecture 91

Data Architecture Architecture Management Banking 91

Netflix Tech

OCTOBER 8, 2024

Building on these foundational abstractions, we developed the TimeSeries Abstraction  — a versatile and scalable solution designed to efficiently store and query large volumes of temporal event data with low millisecond latencies, all in a cost-effective manner across various use cases. For example: {“device_type”: “ios”}.

Bytes 96

Bytes Datasets Metadata Data 96

Lyft Engineering

NOVEMBER 29, 2023

It also allowed us to optimize for handling time-series data and event data at scale. Druid leverages the concept of segments , a unit of storage that allows for parallel querying and columnar storage, complemented with efficient compression and data retrieval. An example of how we use Druid rollup at Lyft.

Kafka 105

Kafka Data Ingestion Datasets Architecture 105

ProjectPro

JANUARY 25, 2022

PySpark is a handy tool for data scientists since it makes the process of converting prototype models into production-ready model workflows much more effortless. PySpark is used to process real-time data with Kafka and Streaming, and this exhibits low latency. RDD uses a key to partition data into smaller chunks.

Big Data 52

Big Data Data Process Process Kafka 52

Towards Data Science

FEBRUARY 9, 2023

An experiment on BigQuery If you are processing a couple of MB or GB with your dbt model, this is not a post for you; you are doing just fine! This post is for those poor souls that need to scan terabytes of data in BigQuery to calculate some counts, sums, or rolling totals over huge event data on a daily or even at a higher frequency basis.

Big Data 82

Big Data Raw Data Aggregated Data Data 82

AltexSoft

DECEMBER 23, 2022

Integrating data from numerous, disjointed sources and processing it to provide context provides both opportunities and challenges. One of the ways to overcome challenges and gain more opportunities in terms of data integration is to build an ELT (Extract, Load, Transform) pipeline. Order of process phases. What is ELT?

Process 52

Process Building Raw Data Data Lake 52

Confluent

MARCH 28, 2019

At Confluent, we want to help developers understand how to think about event streaming and the opportunities it can create. Educating people on what an event stream looks like is a daunting task. Traditionally, making sense of the data flowing in a distributed event streaming platform is done by charts and graphs of aggregated data.

Kafka 74

Kafka Aggregated Data Engineering Media 74

Confluent

MARCH 28, 2019

At Confluent, we want to help developers understand how to think about event streaming and the opportunities it can create. Educating people on what an event stream looks like is a daunting task. Traditionally, making sense of the data flowing in a distributed event streaming platform is done by charts and graphs of aggregated data.

Kafka 79

Kafka Aggregated Data Engineering Media 79

AltexSoft

OCTOBER 21, 2022

Similar to Google in web browsing and Photoshop in image processing, it became a gold standard in data streaming, preferred by 70 percent of Fortune 500 companies. Apache Kafka is an open-source, distributed streaming platform for messaging, storing, processing, and integrating large data volumes in real time.

Kafka 93

Kafka Hadoop Big Data ETL Tools 93

Cloudera

NOVEMBER 13, 2020

Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? It turns out that Apache Impala scales down with data just as well as it scales up. This is part of our series of blog posts on recent enhancements to Impala. The entire collection is available here.

Metadata 144

Metadata Coding SQL Database 144

Snowflake

JUNE 8, 2023

Similarly, we could create a product 360, pulling data from various production processes or software embedded in them, plus sales transactions, service records, and product usage. Those data products could be used by themselves or aggregated into an aggregate data product, like the customer 360 described above.

Aggregated Data 72

Aggregated Data Raw Data Data Telecommunication 72

Netflix Tech

JULY 12, 2019

delivering a large amount of business value in the process. Gatekeeper accomplishes its prescribed task by aggregating data from multiple upstream systems, applying some business logic, then producing an output detailing the status of each video in each country. The team responsible for this curation is Title Operations.

Datasets 73

Datasets Kafka Architecture Computer Science 73

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Arjun Mohnot , Jenchang Ho , Anthony Quigley , Xing Lin , Anil Alluri , Michael Kuchenbecker LinkedIn operates one of the world’s largest Apache Hadoop big data clusters. These SSH-based processes consumed resources, negatively impacting our server and service performance.

Big Data 92

Big Data Hadoop Metadata Data 92

ProjectPro

FEBRUARY 8, 2023

In 2023, more than 5140 businesses worldwide have started using AWS Glue as a big data tool. For e.g., Finaccel, a leading tech company in Indonesia, leverages AWS Glue to easily load, process, and transform their enterprise data for further processing. AWS Glue automates several processes as well.

AWS 98

AWS Metadata Scala Data Lake 98

Rockset

AUGUST 25, 2021

With Confluent’s recent IPO, streaming data has officially gone mainstream, “becoming the underpinning of a modern digital customer experience, and the key to driving intelligent, efficient operations” to quote from their letter to shareholders. Batch processes simply don’t cut it.

Aggregated Data 52

Aggregated Data Hadoop SQL Data Lake 52

Striim

JULY 31, 2024

Why Striim Stands Out As detailed in the GigaOm Radar Report, Striim’s unified data integration and streaming service platform excels due to its distributed, in-memory architecture that extensively utilizes SQL for essential operations such as transforming, filtering, enriching, and aggregating data.

Aggregated Data 52

Aggregated Data Data Ingestion Java Kafka 52

Rockset

JULY 21, 2022

All interactions are streamed in the form of semi-structured events into Firebase’s NoSQL cloud database, where the data, which includes a large number of nested objects and arrays, is ingested. Since we no longer have to set up schemas in advance, we can ingest real-time event streams without interruption into Rockset.

NoSQL 52

NoSQL IT MongoDB SQL 52

Striim

AUGUST 22, 2024

For instance, continuous real-time analytics can be leveraged to analyze streams of network security data flowing into an organization’s network. In addition to the main types of real-time analytics, streaming analytics also plays a crucial role in processing data as it flows in real-time.

Data Warehouse 52

Data Warehouse Retail Machine Learning Database 52

Rockset

AUGUST 30, 2021

However, for many use cases at huge volumes — such as a Kafka topic that streams tens of TBs of data every day — it becomes prohibitively expensive to index the raw data stream and then calculate the desired metrics downstream at query processing time. You can also optionally use WHERE clauses to filter out data.

SQL 52

SQL Kafka MongoDB MySQL 52

Picnic Engineering

OCTOBER 31, 2023

Installations and app instrumentation While installing the Datadog agent on a Kubernetes cluster using their official Helm chart is a straightforward process, configuring application pods can appear rather complex. The capability to aggregate data in one place, combined with a wide range of integrations, simplifies data collection and access.

Java 52

Java Aggregated Data Coding Python 52

Cloudera

NOVEMBER 8, 2017

Event prediction. Building a full customer 360 requires aggregating data sets into a single view. This is a huge step in helping reduce fraud in mobile commerce given that DOCOMO Digital processes more than €3 billion in transactions annually. You can also read about Cloudera Data Science and Engineering here.

Machine Learning 40

Machine Learning Aggregated Data Algorithm Data Science 40

Lyft Engineering

SEPTEMBER 27, 2022

The very first version (see Figure 1) was designed to consume events, convert data to ML features, orchestrate model executions, and sync decision variables to their respective services. This pipeline ingests tens of millions of events per second and processes them into machine learning features.

Kafka 52

Kafka Aggregated Data Machine Learning Architecture 52

Precisely

MARCH 12, 2024

Here’s what the data enrichment process looks like: Aggregating data from a variety of sources Putting the data through ETL processes to ensure they’re useful and clean Appending contextual information to your existing data There are two ways to put these processes into action: manually or through automation.

Insurance 52

Insurance Telecommunication High Quality Data Retail 52

Rockset

FEBRUARY 16, 2024

Furthermore, Rockset’s ability to pre-aggregate data at ingestion time reduced the cost of storage and sped up queries, making the solution cost-effective at scale. With Rockset’s flexible data model , the team could easily define new metrics, add new data and onboard customers without significant engineering resources.

Architecture 59

Architecture SQL Data Warehouse Database 59

LinkedIn Engineering

DECEMBER 20, 2023

Collecting signals is not just about quantity; it's about ensuring the diversity and quality of the data, and ensuring that the data is relevant, accurate, and reflective of the real-world activities of the member. We need systems to ingest, process, and analyze the data efficiently. Espresso , Venice , Rest.li

Building 91

Building Algorithm Kafka Machine Learning 91

Striim

OCTOBER 11, 2024

Then, we’ll explore a data pipeline example and dive deeper into the key differences between a traditional data pipeline vs ETL. What is a Data Pipeline? A data pipeline refers to a series of processes that transport data from one or more sources to a destination, such as a data warehouse, database, or application.

Data Pipeline 52

Data Pipeline MongoDB Unstructured Data Data Lake 52

Knowledge Hut

JANUARY 19, 2024

Tools Used TIBCO PowerBI SAP Business Objects QlikSense Word processing MS Visio MS Office Tools Google docs Approach Business intelligence focuses on descriptive statistics. Business Intelligence v s Business Analytics: Definitions Business Intelligence refers to the process of gathering and analyzing data to make better business decisions.

Business Intelligence 52

Business Intelligence BI Business Analyst Aggregated Data 52

Rockset

MARCH 1, 2023

As a result, compute contention ensues, causing several problems for customers and prospects: User-facing analytics in my SaaS application can only update every 30 minutes since the underlying database becomes unstable whenever I try to process streaming data continuously.

Architecture 59

Architecture AWS SQL Cloud Storage 59

Picnic Engineering

SEPTEMBER 8, 2022

Quick re-cap: the purpose of the internal pipeline is to deliver data from dozens of Picnic back-end services such as warehousing, machine learning models, customers and order status updates. The data is loaded into Snowflake, Picnic’s single source of truth Data Warehouse (DWH). Yet, some messages are destined for the DWH only.

Kafka 52

Kafka Metadata AWS Java 52

Rockset

NOVEMBER 1, 2022

Step 1: Data Acquisition Elasticsearch is rarely the system of record which means the data in it comes from somewhere else for real-time analytics. Rockset has built-in connectors to stream real-time data for testing and simulating production workloads including Apache Kafka , Kinesis and Event Hubs.

Database-centric 52

Database-centric SQL Pipeline-centric Aggregated Data 52

ProjectPro

OCTOBER 28, 2015

Apache Hadoop is synonymous with big data for its cost-effectiveness and its attribute of scalability for processing petabytes of data. Data analysis using hadoop is just half the battle won. Getting data into the Hadoop cluster plays a critical role in any big data deployment.

ETL Tools 52

ETL Tools Hadoop Relational Database Unstructured Data 52

Striim

JULY 10, 2024

By leveraging predictive analytics, logistics companies can optimize supply chain processes, enhance customer satisfaction, and achieve significant cost savings. From forecasting demand to managing operational risks, predictive analytics provides invaluable insights that empower organizations to make data-driven decisions in real-time.

Management 52

Management Transportation Machine Learning High Quality Data 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. In the event that they are not the same, what are the difference s? To provide meaningful business insights, it collects and manages data from a variety of sources. Data Warehouse in DBMS: .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

DareData

JULY 5, 2023

In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently. To overcome this challenge, many companies are turning to Data Lake solutions, which provide a centralized and scalable platform for storing, processing, and analyzing data.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Ascend.io

SEPTEMBER 14, 2023

The rise of data-intensive operations has positioned data engineering at the core of today’s organizations. As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand.

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Rockset

DECEMBER 17, 2020

Overview of the Customer 360 App Our app will make use of real-time data on customer orders and events. We’ll use Rockset to get data from different sources and run analytical queries that power our app in Retool. From there, we’ll create a data API for the SQL query we write in Rockset.

Building 40

Building Aggregated Data SQL Database 40

Databand.ai

JULY 10, 2023

Data analysis: Processing and studying the collected data to recognize patterns, trends, and irregularities that can aid in diagnosing issues or boosting performance. Security: Observability platforms often include built-in security features to ensure the integrity and confidentiality of your data.

Data Pipeline 52

Data Pipeline Algorithm Raw Data Data Engineering 52