Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). In this blog post, we’ll see how you can use other CDP services with COD. Integrated across the Enterprise Data Lifecycle . Cloudera Data Engineering to ingest bulk data and data from mainframes.

Database 83

Database Machine Learning Kafka Aggregated Data 83

Lyft Engineering

APRIL 21, 2023

In a previous blog post , we explored the architecture and challenges of the platform. In our previous blog , we discussed the various challenges we faced in model monitoring and our strategy to address some of these issues. We briefly discussed using z-scores to find anomalies.

Systems 109

Systems Building Machine Learning Raw Data 109

Cloudera

OCTOBER 11, 2022

Most AI apps and ML models need different types of data – real-time data from devices, equipment, and assets and traditional enterprise data – operational, customer, service records. . But it isn’t just aggregating data for models. Data needs to be prepared and analyzed.

Data Science 94

Data Science Aggregated Data Data Consulting 94

Snowflake

APRIL 9, 2024

To improve go-to-market (GTM) efficiency, Snowflake created a bi-directional data share with Outreach that provides consistent access to the current version of all our customer engagement data. In this blog, we’ll take a look at how Snowflake is using data sharing to benefit our SDR teams and marketing data analysts.

BI 104

BI Data Ingestion Data Aggregated Data 104

Workfall

SEPTEMBER 18, 2023

Pair this with Snowflake , the cloud data warehouse that acts as a vault for your insights, and you have a recipe for data-driven success. Get ready to explore the realm where data dreams become reality! In this blog, we will cover: What is Airbyte? With Airbyte and Snowflake, data integration is now a breeze.

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Cloudera

JANUARY 14, 2021

This blog is the final post of a 4-part series. You can read the first blog posts, here: 1. This retailer deployed Cloudera DataFlow to tap real-time streaming data from thousands of cold storage sensors across its vast network of brick-and-mortar stores. Get to Know Your Retail Customer: 2.

Food 72

Food Retail Aggregated Data Machine Learning 72

Cloudera

JUNE 29, 2023

Streamline KYC and AML, too While Know Your Customer (KYC) and Anti-Money-Laundering (AML) processes didn’t play a role in the recent collapses, institutions can also leverage the combination of a modern, open data architecture, advanced analytics, and machine automation to transform KYC and AML.

Data Architecture 85

Data Architecture Architecture Management Banking 85

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? It turns out that Apache Impala scales down with data just as well as it scales up. The entire collection is available here.

Metadata 143

Metadata Coding SQL Database 143

Netflix Tech

DECEMBER 10, 2018

Secondly, we utilize various signals and aggregate data such as understanding of content popularity on Netflix to enable highly relevant ads. Monet helps drive incremental conversions, engagement with our product and in general, present a rich story about our content and the Netflix brand to users around the world.

Coding 111

Coding Aggregated Data Utilities Architecture 111

Netflix Tech

NOVEMBER 20, 2023

In this blog post, we talk about the landscape and the challenges in workflows at Netflix. Downstream workflows (if there is no business logic change) will be triggered by the data change due to backfill. This enables auto propagation of backfill data in multi-stage pipelines.

Process 87

Process Data Pipeline Datasets SQL 87

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Lyft Engineering

JUNE 28, 2023

Our goal was to develop foundations that would enable the hundreds of ML developers at Lyft to efficiently develop new models and enhance existing models with streaming data. In this blog post, we will discuss what we built in support of that goal and some of the lessons we learned along the way.

Machine Learning 120

Machine Learning Building Kafka Metadata 120

Edureka

JANUARY 2, 2025

Use Case: 1: Cardinality is necessary for creating data models that aggregate data, such as those used to monitor product sales, client interactions, or order histories. Many-to-many tables are appropriate when examining non-aggregation among them where no unique keys exist in both tables.

BI 52

BI Certification Database Aggregated Data 52

Cloudera

FEBRUARY 9, 2023

The sudden failing of a complex data pipeline can lead to devastating consequences — especially if it goes unnoticed. This is why we build job notifications functionality into SSB, to deliver maximum reliability in your complex real-time data pipelines.

SQL 79

SQL Kafka Aggregated Data Architecture 79

Data Engineering Weekly

SEPTEMBER 26, 2024

Silver Layer: In this zone, data undergoes cleaning, transformation, and enrichment, becoming suitable for analytics and reporting. Access expands to data analysts and scientists, though sensitive elements should remain masked or anonymized. Grab’s blog on migrating from RBAC to ABAC is an excellent reference design.

Systems 68

Systems Engineering Data Warehouse Architecture 68

Snowflake

FEBRUARY 6, 2023

The latter create integrated, higher-value data products that are geared towards requirements of the data consumers on the business side; for example, a customer 360 domain aggregating data from multiple sources. Some teams are data producers but not data consumers. It’s not just the data teams.

Government 95

Government Data Data Pipeline Architecture 95

Snowflake

JUNE 22, 2023

Types of data products In a previous blog post, I discussed different forms a data product can take using the sand, glass, or lamp metaphor , depending on the consumer of the data and the use case. Some data products will be components in a more complex analysis or context-specific business application.

Building 52

Building Business Analyst Aggregated Data Data 52

Netflix Tech

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Bytes 95

Bytes Datasets Metadata Data 95

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data.

Kafka 107

Kafka Data Ingestion Architecture Datasets 107

LinkedIn Engineering

DECEMBER 20, 2023

In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform. Standing still in the realm of abuse prevention is synonymous with regression.

Building 91

Building Algorithm Kafka Machine Learning 91

Databand.ai

JULY 10, 2023

Faster issue diagnosis: Aggregating data from multiple sources enables engineers to correlate events more easily when troubleshooting problems, allowing them to resolve issues more quickly and prevent future occurrences through proactive measures such as capacity planning or automated remediation actions based on observed trends.

Data Pipeline 52

Data Pipeline Algorithm Data Engineer Data Engineering 52

Cloudera

NOVEMBER 8, 2017

Building a full customer 360 requires aggregating data sets into a single view. You can also read about Cloudera Data Science and Engineering here. The post Machine Learning, the DOCOMO Digital way: Two Core Use Cases appeared first on Cloudera Blog. Driving customer insights with machine learning.

Machine Learning 40

Machine Learning Aggregated Data Algorithm Data Science 40

Rockset

AUGUST 30, 2021

The latest Rockset release, SQL-based rollups, has made real-time analytics on streaming data a lot more affordable and accessible. Anyone who knows SQL, the lingua franca of analytics, can now rollup, transform, enrich and aggregate real-time data at massive scale. You can also optionally use WHERE clauses to filter out data.

SQL 52

SQL Kafka MongoDB MySQL 52

Rockset

NOVEMBER 1, 2022

This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. In this blog, we distilled their migration journeys into 5 steps. We often see ingest queries aggregate data by time.

Database-centric 52

Database-centric SQL Pipeline-centric Aggregated Data 52

LinkedIn Engineering

DECEMBER 19, 2023

The version drift framework consolidates data from various sources, as shown in Figure 6, to create a comprehensive list of worker nodes currently running outdated versions. This framework operates on the scheduler, periodically polls relevant metrics, aggregates data, and determines which nodes have drifted.

Big Data 92

Big Data Hadoop Metadata Data 92

DoorDash Engineering

APRIL 12, 2023

Challenges of ad-hoc SQLs Our initial goal with Curie was to standardize the analysis methodologies and simplify the experiment analysis process for data scientists. Acknowledgements Special thanks to Sagar Akella, Bhawana Goel, and Ezra Berger for their invaluable help in reviewing this blog article.

SQL 83

SQL Metadata Raw Data Government 83

ProjectPro

JANUARY 24, 2023

One of the core features of ADF is the ability to preview your data while creating your data flows efficiently and to evaluate the outcome against a sample of data before completing and implementing your pipelines. Such features make Azure data flow a highly popular tool among data engineers.

Retail 52

Retail Big Data Data Pipeline Media 52

Scott Logic

JULY 26, 2023

My colleague Oliver Cronk has set out some of the principal risks in this blog post. In this blog post, I’ll provide an overview of how it worked. They can’t ask the bot to expose data on other customers or anything else outside of the Knowledge Graph’s purview. So, how do you mitigate the risks and harness the potential?

Banking 52

Banking Aggregated Data Retail Architecture 52

Precisely

MARCH 12, 2024

That’s where data enrichment comes into the picture. In this blog post, we’ll explain what data enrichment is, why you need it, how it works, and how B2B companies can use enriched data to drive results. What is data enrichment? How does data enrichment work?

Insurance 52

Insurance Telecommunication High Quality Data Retail 52

DareData

JULY 5, 2023

In this blog post, we aim to share practical insights and techniques based on our real-world experience in developing data lake infrastructures for our clients - let's start! The Data Lake acts as the central repository for aggregating data from diverse sources in its raw format.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Edureka

FEBRUARY 6, 2023

Azure Data Engineers are in high demand due to the growth of cloud-based data solutions. In this article, we will examine the duties of an Azure Data Engineer as well as the typical pay in this industry. Conclusion So this was all about the salary, job description, and skills of an Azure Data Engineer.

Data Engineer 52

Data Engineer Data Engineering Engineering Amazon Web Services 52

Netflix Tech

APRIL 26, 2022

The Iceberg table created by Keystone contains large blobs of unstructured data. These large unstructured blogs are not efficient for querying, so we need to transform and store this data in a different format to allow efficient queries. Was data corrupted at rest? Compute applications follow daily trends.

Metadata 78

Metadata Datasets Machine Learning Designing 78

U-Next

SEPTEMBER 7, 2022

As training data increases, deep learning requires scalability. Typically, data warehouses are set to read-only for analysts, who primarily read and aggregate data. It is not necessary to insert or update data since it is already clean and archival. . Conclusion . .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Knowledge Hut

JANUARY 25, 2024

It doesn't matter if you're a data expert or just starting out; knowing how to clean your data is a must-have skill. The future is all about big data. This blog is here to help you understand not only the basics but also the cool new ways and tools to make your data squeaky clean.

Data Cleanse 52

Data Cleanse Datasets Data Preparation Data Science 52

Rockset

JULY 29, 2020

The main strategy to alleviate some of the pressure on the primary database is to offload some of the work to a secondary data store, and I will share some of the common patterns of this strategy in this blog series. In future articles I will discuss offloading to other types of systems.

MongoDB 40

MongoDB Database Aggregated Data Accessibility 40

Rockset

DECEMBER 17, 2020

Together, they empower developers to build performant internal tools, such as customer 360 and logistics monitoring apps, by solely using data APIs and pre-built UI components. In this blog, we’ll be building a customer 360 app using Rockset and Retool. From there, we’ll create a data API for the SQL query we write in Rockset.

Building 40

Building SQL Aggregated Data Database 40

Knowledge Hut

OCTOBER 13, 2023

Aggregate Data: If you don't need granularity, consider aggregating data before loading it into Power BI to reduce the volume of data. Sort and Filter Early: Apply sorting and filtering in your queries as early as possible to reduce the amount of data transferred and processed.

BI 52

BI Business Analyst Datasets Raw Data 52