Cloudera

FEBRUARY 8, 2021

To accomplish this, ECC is leveraging the Cloudera Data Platform (CDP) to predict events and to have a top-down view of the car’s manufacturing process within its factories located across the globe. . Having completed the Data Collection step in the previous blog, ECC’s next step in the data lifecycle is Data Enrichment.

Data Pipeline 118

Data Pipeline Building Manufacturing Data Warehouse 118

The Pragmatic Engineer

OCTOBER 17, 2024

A €150K ($165K) grant, three people, and 10 months to build it. Storing data: data collected is stored to allow for historical comparisons. Databases:  SQLite files used to publish data Duck DB to query these files in the public APIs Cockroach DB : used to collect and store historical data.

Cloud 332

Cloud AWS Metadata Cloud Computing 332

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. What technical systems are required to generate and collect those interactions? When is Snowplow the wrong choice?

Building 130

Building IT Metadata MongoDB 130

Snowflake

OCTOBER 9, 2023

Easily collect and store digital events directly to create a complete composable customer data platform (CDP) Marketers are increasingly leveraging the Snowflake Data Cloud as the foundation for all of their customer data analytics and activation. Personalization API : Fetch Data Cloud data for real-time personalization.

Cloud 125

Cloud Building SQL Data 125

Pinterest Engineering

SEPTEMBER 20, 2023

Our commitment is evidenced by our history of building products that champion inclusivity. We know from experience that building for marginalized communities helps make the product work better for everyone. Signal Development and Indexing The process of developing our visual body type signal essentially begins with data collection.

Building 109

Building Pipeline-centric Machine Learning Datasets 109

Netflix Tech

OCTOBER 19, 2020

This insight led us to build Edgar: a distributed tracing infrastructure and user experience. Troubleshooting a session in Edgar When we started building Edgar four years ago, there were very few open-source distributed tracing systems that satisfied our needs. The following sections describe our journey in building these components.

Building 95

Building Transportation Java Metadata 95

AltexSoft

MAY 12, 2022

Audio data transformation basics to know. Before diving deeper into processing of audio files, we need to introduce specific terms, that you will encounter at almost every step of our journey from sound data collection to getting ML predictions. One of the largest audio data collections is AudioSet by Google.

Machine Learning 122

Machine Learning Building Deep Learning Healthcare 122

Edureka

JULY 18, 2024

The secret sauce is data collection. Data is everywhere these days, but how exactly is it collected? This article breaks it down for you with thorough explanations of the different types of data collection methods and best practices to gather information. What Is Data Collection?

Data Collection 40

Data Collection Media Data Science Government 40

AltexSoft

JUNE 26, 2023

While today’s world abounds with data, gathering valuable information presents a lot of organizational and technical challenges, which we are going to address in this article. We’ll particularly explore data collection approaches and tools for analytics and machine learning projects. What is data collection?

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

DoorDash Engineering

APRIL 23, 2024

Product attributes allow DoorDash to group products based on commonalities, building a product profile for each customer around their affinities to certain attributes. These are the building blocks for providing highly relevant and personalized shopping recommendations. Better personalization.

Building 91

Building Retail Manufacturing Unstructured Data 91

Cloudera

DECEMBER 6, 2024

That kind of information is going to become very valuable, and people are going to bid and build markets against that. Data collectives are going to merge over time, and industry value chains will consolidate and share information. It’s not direct competitors. Retail manufacturing distribution is a natural value chain.

Manufacturing 82

Manufacturing Retail Finance Metadata 82

Cloudera

JUNE 17, 2022

In the second blog of the Universal Data Distribution blog series , we explored how Cloudera DataFlow for the Public Cloud (CDF-PC) can help you implement use cases like data lakehouse and data warehouse ingest, cybersecurity, and log optimization, as well as IoT and streaming data collection.

Data Pipeline 94

Data Pipeline Building Kafka Java 94

Snowflake

MARCH 12, 2025

Legacy systems further complicate the situation, as outdated technologies lack the agility and data-sharing capabilities necessary for secure, seamless data collaboration across systems. Adding to the complexity are evolving data privacy regulations , requiring careful, secure use of fan data.

Media 80

Media Cloud Programming Data Collection 80

ThoughtSpot

OCTOBER 18, 2024

In the fast-paced world of software development, the efficiency of build processes plays a crucial role in maintaining productivity and code quality. At ThoughtSpot , while Gradle has been effective, the growing complexity of our projects demanded a more sophisticated approach to understanding and optimizing our builds.

Building 52

Building Process Pipeline-centric Database-centric 52

Data Engineering Podcast

OCTOBER 8, 2023

Max Cho found that fact frustrating enough that he decided to build a business of making policy selection more navigable. In this episode he shares his journey of data collection and analysis and the challenges of automating an intentionally manual industry. Check out the agenda and register today at Neo4j.com/NODES.

Insurance 162

Insurance BI SQL Machine Learning 162

Netflix Tech

FEBRUARY 14, 2025

We will explore the challenges we encounter and unveil how we are building a resilient solution that transforms these client-side impressions into a personalized content discovery experience for every Netflixviewer. The data collected feeds into a comprehensive quality dashboard and supports a tiered threshold-based alerting system.

Kafka 66

Kafka Datasets Metadata Utilities 66

Cloudera

JANUARY 20, 2021

The data journey is not linear, but it is an infinite loop data lifecycle – initiating at the edge, weaving through a data platform, and resulting in business imperative insights applied to real business-critical problems that result in new data-led initiatives. Data Collection Challenge. Factory ID.

Manufacturing 110

Manufacturing Data Warehouse Kafka Retail 110

Engineering at Meta

JANUARY 27, 2023

Data center networking: Over the past decade, on the physical front, we have seen a rise in vendor-specific hardware that comes with heterogeneous feature and architecture sets (e.g., They present key ideas underpinning the FBOSS model that helped them build a stable and scalable network. non-blocking architecture).

Engineering 130

Engineering Software Engineer Software Engineering Transportation 130

Engineering at Meta

APRIL 17, 2023

How it works: Millisampler comprises userspace code to schedule runs, store data, and serve data, and an eBPF-based tc filter that runs in the kernel to collect fine-timescale data. The user code attaches the tc filter and enables data collection.

Bytes 115

Bytes Transportation Data Collection Coding 115

Cloudera

APRIL 9, 2021

This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The first blog introduced a mock vehicle manufacturing company, The Electric Car Company (ECC) and focused on Data Collection.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Christophe Blefari

SEPTEMBER 15, 2023

— Hugo propose 7 hacks to optimise data warehouse cost. ❤️ The key to building a high-performing data team is structured onboarding — The title say it all. How to reduce warehouse costs? Still in the article it mentions 2 key piece.

Data Warehouse 130

Data Warehouse Data SQL Python 130

Cloudera

JUNE 2, 2022

One of the critical requirements that has materialized is the need for companies to take control of their data flows from origination through all points of consumption both on-premise and in the cloud in a simple, secure, universal, scalable, and cost-effective way.

Systems 104

Systems Data Lake Google Cloud Data Collection 104

Cloudera

MAY 4, 2022

The availability and maturity of automated data collection and analysis systems is making it possible for businesses to implement AI across their entire operations to boost efficiency and agility. But you’ll need efficient, intelligent systems such as the Cloudera Data Platform to execute the strategy.

Insurance 130

Insurance Retail Finance Medical 130

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently. Data Sources: How different are your data sources?

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Data Engineering Podcast

NOVEMBER 20, 2022

He explains the constraints that he and his team are faced with and the various challenges that they have overcome to build useful data products on top of a legacy platform where they don’t control the end-to-end systems. Can you describe what League of Legends is and the role that data plays in the experience?

Systems 130

Systems Metadata Data Pipeline MongoDB 130

The Modern Data Company

APRIL 15, 2024

With this in mind, let’s explore how to demystify the process of building your data-driven strategy, making it accessible and actionable. We’ll uncover how you can transform data into a strategic asset that propels your organization forward without getting lost in the complexity of its creation. It matters a lot.

Building 52

Building Raw Data Food Data 52

Striim

MARCH 21, 2025

Take a streaming-first approach to data integration The first, and most important decision is to take a streaming first approach to integration. This means that at least the initial collection of all data should be continuous and real-time.

Process 52

Process Data Warehouse Kafka Data Pipeline 52

Edureka

JANUARY 27, 2025

These tools help in tasks like data collection, reconnaissance, vulnerability detection, and exploitation. Some common tools used by red teams include: Data Collection and Reconnaissance Tools : Red teams often begin by gathering open-source information to understand the target environment.

Media 52

Media Certification Education Programming 52

Ascend.io

JANUARY 2, 2024

But let’s be honest, creating effective, robust, and reliable data pipelines, the ones that feed your company’s reporting and analytics, is no walk in the park. From building the connectors to ensuring that data lands smoothly in your reporting warehouse, each step requires a nuanced understanding and strategic approach.

Data Pipeline 52

Data Pipeline Building Raw Data Data Warehouse 52

Snowflake

JANUARY 18, 2024

Built-in automation eliminates the need for customers to build indexes or do housekeeping. Manufacturing companies no longer need specialists with proprietary programming experience to build queries because users can construct queries using familiar programming constructs.

Manufacturing 111

Manufacturing Unstructured Data Cloud Architecture 111

RandomTrees

NOVEMBER 25, 2024

Solution: Generative AI-Driven Customer Insights In the project, Random Trees, a Generative AI algorithm was created as part of a suite of models for data mining the patterns from patterns in data collections that were too large for traditional models to easily extract insights from.

Telecommunication 52

Telecommunication IT Unstructured Data Data Mining 52

Data Engineering Podcast

FEBRUARY 24, 2020

Summary Misaligned priorities across business units can lead to tensions that drive members of the organization to build data and analytics projects without the guidance or support of engineering or IT staff. What are the benefits to the organization of individuals or teams building and managing their own solutions?

IT 100

IT Data Lake Data Pipeline Media 100

Cloudera

AUGUST 12, 2021

The report classified employees’ reasons for leaving into six broad categories such as growth opportunity and job security, demonstrating the importance of using performance data, data collected from voluntary departures and historical data to reduce attrition for strong performers and enhance employees’ well-being.

Manufacturing 111

Manufacturing Insurance Consulting Education 111

Cloudera

NOVEMBER 19, 2020

Inordinate time and effort are devoted to cleaning and preparing data, resulting in data bottlenecks that impede effective use of anomaly detection tools. A platform approach offers government entities a solid infrastructure upon which to build their fraud prevention and detection efforts. A better approach is needed.

Government 119

Government Machine Learning Algorithm Raw Data 119

Data Engineering Podcast

JULY 20, 2020

I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems 100

Systems Machine Learning Manufacturing Algorithm 100

Snowflake

MARCH 7, 2023

For example, utilizing data infrastructures that can scale compute resources up and down to handle fluctuating demand will inherently be more energy efficient than a data warehouse with regimented sizing. You should use the data you already have. Data collection and disclosure requirements keep shifting.

Transportation 100

Transportation Data Data Lake Data Collection 100

Monte Carlo

MAY 31, 2023

And in the same way that no two organizations are identical, no two data integrity frameworks will be either. On the other hand, healthcare organizations with strict compliance standards related to sensitive patient information might require a completely different set of data integrity processes to maintain internal and external standards.

Building 52

Building Data Validation Healthcare Data Integration 52