ThoughtSpot

APRIL 22, 2025

Level 2: Understanding your dataset To find connected insights in your business data, you need to first understand what data is contained in the dataset. Spotter quickly translates your datasets into business-friendly terminology so business users can confidently explore their data through natural language conversations.

BI 59

BI Datasets Business Intelligence Raw Data 59

DataKitchen

NOVEMBER 5, 2024

This architecture is valuable for organizations dealing with large volumes of diverse data sources, where maintaining accuracy and accessibility at every stage is a priority. The Silver layer aims to create a structured, validated data source that multiple organizations can access. How do you ensure data quality in every layer ?

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

Netflix Tech

FEBRUARY 14, 2025

In this multi-part blog series, we take you behind the scenes of our system that processes billions of impressions daily. Architecture Overview The first pivotal step in managing impressions begins with the creation of a Source-of-Truth (SOT) dataset.

Kafka 66

Kafka Datasets Metadata Utilities 66

Picnic Engineering

APRIL 7, 2025

A little over a year ago, we shared a blog post about our journey to enhance customers meal planning experience with personalized recipe recommendations. The approach struggled with scalability , making it difficult to handle large datasets efficiently. These connections form a dataset of interactions.

Datasets 52

Datasets Systems Architecture Machine Learning 52

phData: Data Engineering

NOVEMBER 8, 2024

The world we live in today presents larger datasets, more complex data, and diverse needs, all of which call for efficient, scalable data systems. In this blog, we will discuss: What is the Open Table format (OTF)? These formats are transforming how organizations manage large datasets. Why should we use it?

Architecture 52

Architecture Systems Data Lake Google Cloud 52

Pinterest Engineering

NOVEMBER 18, 2024

In this blog post, we’ll explore what CDC is, why it’s important, and our journey of implementing Generic CDC solutions for all online databases at Pinterest. This is particularly useful in environments where multiple applications need to access and process the same data. What is Change Data Capture? or its affiliates.

Kafka 50

Kafka MySQL Database Software Engineer 50

Engineering at Meta

APRIL 28, 2025

Each product features its own distinct data model, physical schema, query language, and access patterns. Machine learning models : trained on labeled datasets using supervised learning and improved through unsupervised learning to identify patterns and anomalies in unlabeled data.

Metadata 67

Metadata Data Utilities Data Warehouse 67

Ascend.io

NOVEMBER 14, 2024

In this blog post, we’ll explore key strategies for future-proofing your data pipelines. These platforms enable scalable and distributed data processing, allowing data teams to efficiently handle massive datasets. Access Controls Access controls are another critical component of data pipeline security.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Integration Data 52

Cloudyard

MARCH 6, 2025

However, a common challenge arises: Hardcoded role names in masking policies make managing access permissions cumbersome. This blog presents a dynamic, scalable, and efficient approach to role-base data masking in Snowflake using JavaScript UDFs. Not scalable Managing multiple policies across different datasets is tedious.

Accessible 52

Accessible Accessibility Datasets Data Security 52

Data Engineering Weekly

FEBRUARY 9, 2025

MoEs necessitate less compute for pre-training compared to dense models, facilitating the scaling of model and dataset size within similar computational budgets. I found the product blog from QuantumBlack gives a view of data quality in unstructured data.

Data Engineering 58

Data Engineering Data Engineer Engineering Unstructured Data 58

Pinterest Engineering

APRIL 7, 2025

In this blog post, well discuss our experiences in identifying the challenges associated with EC2 network throttling. For these use cases, typically datasets are generated offline in batch jobs and get bulk uploaded from S3 to the database running on EC2. 4xl with up to 12.5

AWS 57

AWS Bytes Database Data Ingestion 57

Edureka

MARCH 15, 2025

By learning the details of smaller datasets, they better balance task-specific performance and resource efficiency. It is seamlessly integrated across Meta’s platforms, increasing user access to AI insights, and leverages a larger dataset to enhance its capacity to handle complex tasks. What are Small language models?

Entertainment 74

Entertainment Retail Education Healthcare 74

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 113

Data Science Cloud Hadoop Metadata 113

Netflix Tech

JANUARY 2, 2025

However, due to the absence of a control group in these countries, we adopt a synthetic control framework ( blog post ) to estimate the counterfactual scenario. To facilitate easier access to incrementality results, we have developed an interactive tool powered by this framework. Data quality plays a huge role in this work.

Engineering 76

Engineering Entertainment Designing Technology 76

Ascend.io

OCTOBER 28, 2024

In this blog post, we’ll explore fundamental concepts, intermediate strategies, and cutting-edge techniques that are shaping the future of data engineering. Filling in missing values could involve leveraging other company data sources or even third-party datasets.

Raw Data 52

Raw Data Datasets Aggregated Data Data Pipeline 52

Ascend.io

DECEMBER 4, 2024

In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Are your tools simple to implement and accessible to users with diverse skill sets? Leveraging cloud-based platforms and distributed computing can help handle large datasets efficiently.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Cloudera

SEPTEMBER 18, 2024

InDaiX provides data consumers with unparalleled flexibility and scalability, streamlining how businesses, researchers, and developers access and integrate diverse data sources and AI foundational models, expediting the process of Generative AI (GenAI) adoption.

Datasets 91

Datasets Retail Healthcare Finance 91

Knowledge Hut

FEBRUARY 29, 2024

Then, based on this information from the sample, defect or abnormality the rate for whole dataset is considered. Hypothesis testing is a part of inferential statistics which uses data from a sample to analyze results about whole dataset or population. It offers various blogs based on above mentioned technology in alphabetical order.

Data Science 98

Data Science Datasets Machine Learning Database Design 98

Data Engineering Weekly

MARCH 5, 2025

This blog post expands on that insightful conversation, offering a critical look at Iceberg's potential and the hurdles organizations face when adopting it. In the mid-2000s, Hadoop emerged as a groundbreaking solution for processing massive datasets. Cost: Reducing storage and processing expenses. Speed: Accelerating data insights.

Hadoop 58

Hadoop Metadata Data Ingestion Data Governance 58

Cloudera

FEBRUARY 22, 2022

The data architecture layer is one such area where growing datasets have pushed the limits of scalability and performance. The data explosion has to be met with new solutions, that’s why we are excited to introduce the next generation table format for large scale analytic datasets within Cloudera Data Platform (CDP) – Apache Iceberg.

Metadata 110

Metadata Datasets BI SQL 110

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology.

Data Pipeline 118

Data Pipeline Building Manufacturing Data Warehouse 118

Cloudera

AUGUST 10, 2021

For example, writing a Spark dataset to Ozone or launching a DDL query in Hive that points to a location in Ozone. or higher with Kerberos enabled and admin access to both Ranger and Atlas. For example, my data volume could contain multiple buckets for every stage of the data, and I can control who accesses each stage.

Hadoop 105

Hadoop Kafka Datasets Government 105

Maxime Beauchemin

AUGUST 28, 2017

Change Management Given that useful datasets become widely used and derived in ways that results in large and complex directed acyclic graphs (DAGs) of dependencies, altering logic or source data tends to break and/or invalidate downstream constructs.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Towards Data Science

JANUARY 27, 2025

Project explanation The dataset for this project was reading data from my personal Goodreads account; it can be downloaded from my GitHub repo. If you use Goodreads, you can export your own data in CSV format, substitute it for the dataset provided, and explore it with the code for this tutorial. build(model).run()

Building 53

Building Datasets Coding Data 53

Pinterest Engineering

JANUARY 4, 2024

This blog post goes into the details of how we built this massively scalable, highly available wide column database using RocksDB, and provides information about the data model, APIs, and key features. It is written in C++ and offers bindings for several programming languages, making it accessible for developers in different environments.

Database 98

Database Building Datasets Relational Database 98

Data Engineering Weekly

AUGUST 6, 2024

CDC enables real-time data flows from operational systems to analytics platforms, ensuring decision-makers can access the most current data. Challenges: Highly resource-intensive and slow for large datasets. However, access to the transaction log is required, which may not always be feasible depending on the database system.

Data Lake 126

Data Lake Data Warehouse Database Data Architecture 126

Cloudera

AUGUST 17, 2022

?. What if you could access all your data and execute all your analytics in one workflow, quickly with only a small IT team? Readily Accessible Data Ingestion and Analytics. Sophisticated data practitioners and business analysts want access to new datasets that can help optimize their work and transform whole business functions.

Cloud Computing 116

Cloud Computing Cloud Storage Data Science Government 116

Ascend.io

DECEMBER 4, 2024

In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. Are your tools simple to implement and accessible to users with diverse skill sets? Leveraging cloud-based platforms and distributed computing can help handle large datasets efficiently.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

Striim

APRIL 18, 2025

Incomplete records : Missing values or partial datasets lead to inaccurate AI predictions and poor business decisions. Why AI and Analytics Require Real-Time, High-Quality Data To extract meaningful value from AI and analytics, organizations need data that is continuously updated, accurate, and accessible.

High Quality Data 52

High Quality Data Business Intelligence Unstructured Data Data Pipeline 52

Edureka

APRIL 14, 2025

This blog breaks down how these tools complement and differ from one another to help you identify the best fit for your business. Broad data connectivity : Seamless integration with numerous data sources enables streamlined access and analysis across systems. Conversely, the reporting tool shines in front-end customization.

BI 40

BI Business Intelligence Raw Data Retail 40

phData: Data Engineering

NOVEMBER 7, 2024

In this blog, we will explore the underlying architecture of transformer-based large vision models, offering insights into how they function. For interested readers looking to learn about transformers applied in the NLP domain, see the What are Large Language Models (LLMs) blog post. Contact phData Today!

Architecture 52

Architecture Project Datasets Utilities 52

Cloudera

NOVEMBER 7, 2023

The vast tapestry of data types spanning structured, semi-structured, and unstructured data means data professionals need to be proficient with various data formats such as ORC, Parquet, Avro, CSV, and Apache Iceberg tables, to cover the ever growing spectrum of datasets – be they images, videos, sensor data, or other type of media content.

Systems 105

Systems Hadoop Unstructured Data Media 105

Netflix Tech

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Bytes 95

Bytes Datasets Metadata Data 95

Cloudera

JANUARY 25, 2022

The explosion of new and more accessible ML tooling means there’s never been a better time to take the leap into predictive analytics than right now. . Democratizing data access breaks down silos and opens insights to any stage of the business operation. Jump start your journey with AMPs. Figure: CML’s Applied ML Prototypes (AMPs).

Datasets 91

Datasets Machine Learning Accessible Accessibility 91

Confluent

JULY 3, 2019

The idea in this blog post is to mix information coming from two distinct channels: the RSS feeds of sport-related newspapers and Twitter feeds of the FIFA Women’s World Cup. For this blog post, we’ll use RSS feeds as a way to measure the average sentiment of the news. wwc : defines the BigQuery dataset name.

Data Analysis 21

Data Analysis Kafka Datasets Java 21

Cloudera

AUGUST 3, 2022

Fine grained access control (FGAC) with Spark. That’s why we are excited to introduce Spark Secure Access , a new security feature for Apache Spark in the Cloudera Data Platform (CDP), that adheres to all security policies without resorting to 3rd party tools. Introducing Spark Secure Access Mode. Starting with CDP 7.1.7

Datasets 90

Datasets Machine Learning BI Accessible 90

Sync Computing

FEBRUARY 4, 2025

What are my most commonly used datasets? Knowing which of your datasets is used the most is crucial for optimizing data access and storage. Go here for the query Data and Workflow Management A couple queries to help manage your data and workflows on Databricks. Where are they stored?

SQL 52

SQL Datasets Utilities Accessibility 52