Cloudera

MAY 19, 2021

In the previous blog post in this series, we walked through the steps for leveraging Deep Learning in your Cloudera Machine Learning (CML) projects. To try and predict this, an extensive dataset including anonymised details on the individual loanee and their historical credit history are included. Get the Dataset. Introduction.

Machine Learning 138

Machine Learning Datasets Data Science Raw Data 138

Team Data Science

MARCH 15, 2020

I am taking you through my recent experience to find a dataset for my project. I defined a few sources in my earlier blog post, which will give a sneak peek of techniques to extract industries. Criteria Define a simple layout to your dataset with elements like size, type of columns, format.

Insurance 130

Insurance Datasets Banking Finance 130

KDnuggets

NOVEMBER 30, 2023

The blog discusses five platforms designed for data scientists with specialized capabilities in managing large datasets, models, workflows, and collaboration beyond what GitHub offers.

Data Science 129

Data Science Project Datasets Data 129

Team Data Science

NOVEMBER 28, 2020

During our first week, Andreas helped us navigate the industry that interests each of us, and this is how we picked a dataset to work on for the next following weeks. I ended up picking yelp dataset , which aligns with my interests in analyzing user behaviors. Stay tuned! Cheers, Liuna Say hello on LinkedIn: [link]

Engineering 130

Engineering Datasets Data Data Engineering 130

Christophe Blefari

APRIL 19, 2024

It was trained on a large dataset containing 15T tokens (compared to 2T for Llama 2). This blog shows how you can use Gen AI to evaluate inputs like translations with added reasons. — A great blog to answer a great question. Llama has a larger tokeniser and the context window grew to 8192 tokens as input.

MySQL 130

MySQL Data Datasets SQL 130

Netflix Tech

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. For more information regarding this, refer to our previous blog.

Datasets 100

Datasets Computer Science Systems Kafka 100

Knowledge Hut

FEBRUARY 29, 2024

Then, based on this information from the sample, defect or abnormality the rate for whole dataset is considered. Hypothesis testing is a part of inferential statistics which uses data from a sample to analyze results about whole dataset or population. It offers various blogs based on above mentioned technology in alphabetical order.

Data Science 98

Data Science Datasets Machine Learning Database Design 98

KDnuggets

JULY 5, 2024

In this blog, we will define Pandas and provide an example of how you can vectorize your Python code to optimize dataset analysis using Pandas to speed up your code over 300x times faster.

Python 100

Python Datasets Coding 100

Cloudera

FEBRUARY 22, 2022

The data architecture layer is one such area where growing datasets have pushed the limits of scalability and performance. The data explosion has to be met with new solutions, that’s why we are excited to introduce the next generation table format for large scale analytic datasets within Cloudera Data Platform (CDP) – Apache Iceberg.

Metadata 111

Metadata Datasets BI SQL 111

Pinterest Engineering

SEPTEMBER 12, 2023

transformers) became standardized, ML engineers started to show a growing appetite to iterate on datasets. While such dataset iterations can yield significant gains, we observed that only a handful of such experiments were conducted and productionized in the last six months. As model architecture building blocks (e.g.

Data Process 104

Data Process Process Datasets Software Engineer 104

Pinterest Engineering

JANUARY 4, 2024

This blog post goes into the details of how we built this massively scalable, highly available wide column database using RocksDB, and provides information about the data model, APIs, and key features. Individual rows constitute a dataset. Row key uniquely identifies a row in a dataset. no in-place modifications are done).

Database 95

Database Building Datasets Relational Database 95

Cloudera

SEPTEMBER 15, 2021

The edge is a critical component of many digital transformation implementations, and particularly IoT deployments, for three main reasons — immediacy, fast-changing datasets and scalability. As Bernard Marr , a futurist and technology consultant, explained in a Cloudera digital event , that today’s datasets have a short shelf life.

Medical 121

Medical Retail Datasets Algorithm 121

Data Engineering Weekly

JULY 7, 2024

The blog highlights the advantages of GNN over traditional machine learning models, which struggle to discern relationships between various entities, such as users and restaurants, and edges, such as order. The blog gives an overview of statistical techniques used in drift detection.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Lake 98

Cloudera

SEPTEMBER 18, 2024

InDaiX is being evaluated as an extension of Cloudera to include: Datasets Exchange: Industry Datasets: Comprehensive datasets across various domains, including healthcare, finance, and retail. Synthetic Datasets: High-quality synthetic data generated using state-of-the-art techniques, ensuring privacy and compliance.

Datasets 93

Datasets Retail Healthcare Finance 93

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. This blog series follows the manufacturing, operations and sales data for a connected vehicle manufacturer as the data goes through stages and transformations typically experienced in a large manufacturing company on the leading edge of current technology.

Data Pipeline 120

Data Pipeline Building Manufacturing Data Warehouse 120

Maxime Beauchemin

AUGUST 28, 2017

Change Management Given that useful datasets become widely used and derived in ways that results in large and complex directed acyclic graphs (DAGs) of dependencies, altering logic or source data tends to break and/or invalidate downstream constructs.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

Data Engineering Weekly

JULY 28, 2024

The blog is an excellent summarization of the common patterns emerging in GenAI platforms. Switching from Apache Spark to Ray improves compact 12X larger datasets than Apache Spark, improves cost efficiency by 91%, and processes 13X more data per hour. Swiggy recently wrote about its internal platform, Hermes, a text-to-SQL solution.

Data Engineering 109

Data Engineering Data Engineer Engineering Database-centric 109

Cloudera

NOVEMBER 13, 2024

Fine Tuning Studio enables users to track the location of all datasets, models, and model adapters for training and evaluation. We can import this dataset on the Import Datasets page. The goal is to train an adapter for this base model that gives it better predictive capabilities for our specific dataset. Model Selection.

Datasets 82

Datasets Machine Learning Coding Data Preparation 82

databricks

SEPTEMBER 24, 2024

This blog post explores how to create a Genie space using a World of Warcraft dataset, enabling users to interactively query data and gain insights like a data analyst. Unlock the potential of your data with Databricks' AI/BI Genie spaces!

BI 88

BI Datasets Data Analysis Entertainment 88

Pinterest Engineering

SEPTEMBER 26, 2023

We have published a detailed blog post of its modeling architecture. Figure 1: hybrid logging for features On a daily basis, the features are joined with the labels to produce the final training dataset. The recommendations are powered by innovative and cutting-edge machine learning technologies.

Software Engineer 104

Software Engineer Software Engineering Machine Learning Datasets 104

databricks

MARCH 26, 2023

In today's business landscape, secure and cost-effective data sharing is more critical than ever for organizations looking to optimize their internal and external.

Datasets 81

Datasets Data 81

Cloudera

AUGUST 10, 2021

For example, writing a Spark dataset to Ozone or launching a DDL query in Hive that points to a location in Ozone. I’ve chosen those names because I’ll be using an easy method for generating and writing TPC-DS datasets, along with creating their corresponding Hive tables. Create a dataset from the customer table. With CDP 7.1.4

Hadoop 106

Hadoop Kafka Datasets Government 106

Data Engineering Weekly

AUGUST 6, 2024

Challenges: Highly resource-intensive and slow for large datasets. Provide mechanisms for handling large datasets efficiently. It can be a blog link or a short note in README. It can be a blog link or a short note in README. This method is often a last resort when other CDC techniques are infeasible.

Data Lake 125

Data Lake Data Warehouse Database Data Architecture 125

AltexSoft

AUGUST 25, 2021

You can’t simply feed the system your whole dataset of emails and expect it to understand what you want from it. Now, when we understand the methodologies and principles behind building NLP models, let’s tackle the main component of all ML projects — a dataset. Preparing an NLP dataset. Determining dataset size.

Process 139

Process Deep Learning Datasets Machine Learning 139

Cloudera

FEBRUARY 1, 2022

In this blog we’ll dig into how the Deep Learning for Image Analysis AMP can be reused to find snowflakes that are less similar to one another. However, because we are only interested in comparing snowflakes, we need to bring our own dataset consisting solely of snowflakes, and a lot of them. Launch the AMP. ICONIC_PATH = “./app/frontend/build/assets/semsearch/datasets/iconic200/”.

Deep Learning 113

Deep Learning Datasets Coding Machine Learning 113

Cloudera

DECEMBER 11, 2020

In a previous blog post on CDW performance, we compared Azure HDInsight to CDW. In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to EMR 6.0 (also powered by Apache Hive-LLAP) on Amazon using the TPC-DS 2.9 More on this later in the blog.

Data Warehouse 111

Data Warehouse Metadata Datasets Data 111

Cloudera

SEPTEMBER 29, 2020

In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to Microsoft HDInsight (also powered by Apache Hive-LLAP) on Azure using the TPC-DS 2.9 A TPC-DS 10TB dataset was generated in ACID ORC format and stored on the ADLS Gen 2 cloud storage. benchmark.

Data Warehouse 122

Data Warehouse Cloud Storage Metadata Cloud 122

Uber Engineering

FEBRUARY 23, 2023

In this blog learn how we automated column-level drift detection in batch datasets at Uber scale, reducing the median time to detect issues in critical datasets by 5X. Data quality is of paramount importance at Uber, powering critical decisions and features.

Systems 84

Systems Datasets Data 84

Netflix Tech

OCTOBER 8, 2024

In previous blog posts, we introduced the Key-Value Data Abstraction Layer and the Data Gateway Platform , both of which are integral to Netflix’s data architecture. Configurability : TimeSeries offers a range of tunable options for each dataset, providing the flexibility needed to accommodate a wide array of use cases.

Bytes 96

Bytes Datasets Metadata Data 96

Christophe Blefari

JANUARY 7, 2024

What 2023 brought: Followers — I doubled in followers on my 3 main platforms: I reached 4000 people on the blog, 8000 on LinkedIn and almost 600 on Twitter (even if I don't post that much there). The blog — 46 articles published in 2023, this is way less than in 2022 but it's ok. It's time.

Data 130

Data SQL Python Data Engineering 130

Cloudera

NOVEMBER 30, 2022

We have divided the “ Transaction Support in Cloudera Operational Database (COD)” blog into two parts. var dataSet = List(Row(1, "1", 1), Row(2, "2", 2)). dataSet = dataSet :+ Row(w, "foo", w); }. var rowRDD = spark.sparkContext.parallelize(dataSet). dataSet = List(Row(501, "500", 500), Row(502, "502", 502)).

Database 84

Database Datasets NoSQL Big Data 84

Data Engineering Weekly

JULY 14, 2024

The blog highlights the reasoning behind selecting dbt and Dagster and some of the key improvements while adopting them, such as handling race conditions in dbt incremental update and bulk backfilling with Dagster. The blog provided a nice comparison summary of various 3rd-party data providers in this space and their capabilities.

Data Engineering 98

Data Engineering Data Engineer Engineering Unstructured Data 98

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. From analyzing your metadata, query logs, and dashboard activities, Select Star will automatically document your datasets.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Cloudera

AUGUST 17, 2021

DE, DW, and ML practitioners that want to orchestrate multi-step data pipelines in the cloud, using a combination of Spark and Hive, can now generate curated datasets for use by downstream applications efficiently and securely. The post Automating Data Pipelines in CDP with CDE Managed Airflow Service appeared first on Cloudera Blog.

Data Pipeline 115

Data Pipeline Management BI Python 115

Engineering at Meta

OCTOBER 3, 2024

The dataset and code are available now on GitHub. Governments in advanced economies can rely on a variety of sources including tax records or census datasets to better estimate their population and make informed decisions on the delivery of services. However, in other parts of the world, accurate population data is hard to come by.

Datasets 86

Datasets Government Banking Programming 86

Christophe Blefari

APRIL 5, 2024

The data analyst every CEO wants — I really like this blog from Benoit, he gives practical advices about what you have to focus on if you're working as an data analyst for C-level of your company. In a nutshell, I'd say that both technologies are not yet mature, with a smaller advantage for Cube for being open.

SQL 130

SQL Data Python Datasets 130

Pinterest Engineering

JULY 25, 2023

Each dataset needs to be securely stored with minimal access granted to ensure they are used appropriately and can easily be located and disposed of when necessary. As businesses grow, so does the variety of these datasets and the complexity of their handling requirements.

Big Data 98

Big Data Accessible Accessibility Hadoop 98