Metadata related to user interactions with reviews, such as ratings and helpful votes, can also play a crucial role in feature representation. Ratings reflect the user’s sentiment or preference towards a particular item, allowing the recommender system to consider user sentiment when generating recommendations. Helpful votes indicate the perceived usefulness of a review by other users, and incorporating this information can help identify influential or trusted reviews for better recommendation accuracy.

Also Try: Human Activity Recognition – Data Science Projects

By leveraging text representation techniques like Bag-of-Words, TF-IDF, and N-grams, and incorporating metadata such as product categories, ratings, and helpful votes, the recommender system gains a comprehensive understanding of the reviews and associated items. This enables the system to capture both the textual content and the context in which the reviews and items exist, leading to more accurate and tailored recommendations for users.

Recommender Systems Using Amazon Reviews: Building Recommender Models

This section will dive into the heart of the project—the construction of recommender models. Collaborative filtering models, including user-based and item-based approaches, will be implemented. We will explore matrix factorization techniques like singular value decomposition (SVD) and non-negative matrix factorization (NMF) to uncover latent factors and generate recommendations. Content-based filtering models will be built, leveraging textual features extracted from reviews. Techniques such as cosine similarity, TF-IDF weighting, and term frequency-inverse document frequency (TF-IDF) will be applied. Hybrid recommender models, combining collaborative and content-based filtering, will be developed, leveraging weighted hybrid models, cascading approaches, and ensemble methods for optimal recommendations.

Collaborative Filtering Models

User-based Collaborative Filtering

User-based collaborative filtering is a widely used technique in recommender systems. It identifies similar users based on their preferences and behaviors and recommends items that those similar users have shown interest in. By finding users with similar tastes, this approach leverages the wisdom of the crowd to generate recommendations. User-based collaborative filtering is effective in situations where user preferences play a significant role in determining recommendations.

Item-based Collaborative Filtering

Item-based collaborative filtering focuses on the similarity between items instead of users. It identifies items that have been interacted with similarly by users and recommends items that are similar to those that a user has already consumed or appreciated. By leveraging item-item similarity, this approach can provide personalized recommendations even in cases where user preferences are less clear or data sparsity is an issue.

Matrix Factorization Techniques (SVD, NMF)

Matrix factorization techniques, such as Singular Value Decomposition (SVD) and Non-Negative Matrix Factorization (NMF), are widely used in collaborative filtering models. These techniques factorize the user-item interaction matrix into lower-dimensional latent factors. By decomposing the matrix, they uncover underlying patterns and relationships between users and items. SVD and NMF help in capturing latent user preferences and item characteristics, enabling accurate recommendations.

Content-based Filtering Models

Cosine Similarity

Cosine similarity is a popular technique used in content-based filtering models. It measures the similarity between two items based on their feature vectors. In the context of recommender systems, feature vectors can represent item attributes such as product descriptions, categories, or keywords. By calculating the cosine similarity between items, this approach identifies items with similar characteristics and recommends items that align with the user’s preferences.

TF-IDF Weighting

TF-IDF (Term Frequency-Inverse Document Frequency) is commonly used in content-based filtering models to weigh the importance of words or features in a document. It assigns higher weights to words that are frequent in a specific document but relatively rare across the entire corpus. By incorporating TF-IDF weighting, content-based models can effectively capture the discriminative power of words and prioritize significant features for recommendation purposes.

Term Frequency-Inverse Document Frequency (TF-IDF)

Term Frequency-Inverse Document Frequency (TF-IDF) is another essential technique in content-based filtering models. It calculates the frequency of a term in a document (term frequency) and adjusts it based on the inverse document frequency. TF-IDF takes into account both the importance of a term in a document and its rarity in the entire corpus. By applying TF-IDF weighting to terms or features, content-based models can emphasize relevant and discriminative aspects for accurate recommendations.

Hybrid Recommender Models

Weighted Hybrid Models

Weighted hybrid models combine collaborative filtering and content-based filtering by assigning weights to recommendations from each approach. These models leverage the strengths of both techniques and provide a more balanced and comprehensive set of recommendations. By considering the preferences derived from collaborative filtering and the relevance determined by content-based filtering, weighted hybrid models aim to enhance recommendation accuracy and coverage.

Cascading Approaches

Cascading approaches in hybrid recommender systems apply a sequential process to generate recommendations. They start with one filtering technique, such as collaborative filtering, to generate an initial set of recommendations. Then, these recommendations are filtered further using a content-based filtering technique. This cascading process refines and improves the recommendations by combining the benefits of both techniques in a controlled manner.

Ensemble Methods

Ensemble methods combine the predictions of multiple recommender models to generate recommendations. These methods leverage the diversity of individual models to overcome biases and enhance recommendation accuracy. By combining the outputs of different models, such as collaborative filtering and content-based filtering, ensemble methods aim to capture a broader range of user preferences and provide robust and diverse recommendations.

By employing collaborative filtering models, content-based filtering models, or hybrid recommender models, developers can leverage different techniques to build effective recommendation systems. These models offer flexibility and customization options to address specific recommendation requirements and cater to the diverse needs of users.

Recommender Systems Using Amazon Reviews: Evaluation and Performance Metrics

To assess the effectiveness of the recommender system, evaluation techniques, and performance metrics are essential. We will explore evaluation methodologies such as holdout, cross-validation, and leave-one-out. Metrics such as precision, recall, F1 score, mean average precision (MAP), and normalized discounted cumulative gain (NDCG) will be employed to measure recommendation quality and effectiveness.

Evaluation Techniques

Holdout

The holdout method is a commonly used evaluation technique in recommender systems. It involves splitting the available data into a training set and a test set. The model is trained on the training set, and its performance is then evaluated on the test set. The holdout method provides a straightforward approach to measure how well the recommender system performs on unseen data.

Cross-Validation

Cross-validation is another evaluation technique that addresses the concern of limited data availability. It involves partitioning the data into multiple subsets or “folds.” The model is trained and evaluated iteratively, with each fold serving as the test set while the remaining folds are used for training. By averaging the results from multiple iterations, cross-validation provides a more robust estimate of the model’s performance.

Leave-One-Out

Leave-One-Out (LOO) is a specialized form of cross-validation where each data point is treated as a separate fold. In each iteration, one data point is left out for testing, and the rest of the data is used for training. LOO provides a comprehensive evaluation by utilizing all available data points as test cases. However, it can be computationally expensive and may not be feasible for larger datasets.

Performance Metrics

Precision, Recall, F1 Score

Precision, recall, and F1 score are commonly used performance metrics for recommender systems:

• Precision measures the proportion of relevant items among the recommended items. It focuses on the accuracy of the recommendations by assessing how many recommended items are truly relevant to the user’s preferences.
• Recall measures the proportion of relevant items that are successfully recommended. It assesses the system’s ability to retrieve all relevant items, capturing the comprehensiveness of the recommendations.
• F1 score is the harmonic mean of precision and recall, providing a balanced measure that considers both precision and recall. It combines the precision and recall values into a single metric, offering a consolidated assessment of the system’s performance.

Mean Average Precision (MAP)

Mean Average Precision (MAP) is a metric commonly used in information retrieval tasks, including recommender systems. It calculates the average precision across different levels of recall. MAP takes into account both precision and the order of recommendations. It is particularly useful when dealing with scenarios where the position or ranking of the recommended items is important.

Normalized Discounted Cumulative Gain (NDCG)

Normalized Discounted Cumulative Gain (NDCG) is a metric that evaluates the ranking quality of the recommendations. It considers both the relevance of the recommended items and their positions in the ranked list. NDCG assigns higher scores to recommendations that are both relevant and ranked higher. It is a valuable metric for assessing the overall quality and effectiveness of the recommendation list.

By utilizing evaluation techniques such as holdout, cross-validation, or leave-one-out, and performance metrics such as precision, recall, F1 score, MAP, and NDCG, developers can assess and compare the performance of recommender systems. These metrics help in understanding how well the system is performing, identifying areas for improvement, and optimizing the recommendation algorithms and strategies.

Recommender System Using Amazon Reviews: Code

You can find the code and dataset on the GitHub page.

1: Import Libraries

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import os
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
import math
import json
import time
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.model_selection import train_test_split
from sklearn.neighbors import NearestNeighbors
from sklearn.externals import joblib
import scipy.sparse
from scipy.sparse import csr_matrix
from scipy.sparse.linalg import svds
import warnings; warnings.simplefilter('ignore')
%matplotlib inline

for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# Any results you write to the current directory are saved as output.

2: Load the Dataset and Add headers

electronics_data=pd.read_csv("/kaggle/input/amazon-product-reviews/ratings_Electronics (1).csv",names=['userId', 'productId','Rating','timestamp'])

# Display the data

electronics_data.head()

#Shape of the data
electronics_data.shape

#Taking subset of the dataset
electronics_data=electronics_data.iloc[:1048576,0:]

#Check the datatypes
electronics_data.dtypes

electronics_data.info()

#Five point summary 

electronics_data.describe()['Rating'].T

#Find the minimum and maximum ratings
print('Minimum rating is: %d' %(electronics_data.Rating.min()))
print('Maximum rating is: %d' %(electronics_data.Rating.max()))

3: Handling Missing values

#Check for missing values
print('Number of missing values across columns: \n',electronics_data.isnull().sum())

# Check the distribution of the rating
with sns.axes_style('white'):
    g = sns.factorplot("Rating", data=electronics_data, aspect=2.0,kind='count')
    g.set_ylabels("Total number of ratings")

4: Unique Users and products

print("Total data ")
print("-"*50)
print("\nTotal no of ratings :",electronics_data.shape[0])
print("Total No of Users   :", len(np.unique(electronics_data.userId)))
print("Total No of products  :", len(np.unique(electronics_data.productId)))

5: Dropping the TimeStamp Column

#Dropping the Timestamp column

electronics_data.drop(['timestamp'], axis=1,inplace=True)

6: Analyzing the rating

#Analysis of rating given by the user 

no_of_rated_products_per_user = electronics_data.groupby(by='userId')['Rating'].count().sort_values(ascending=False)

no_of_rated_products_per_user.head()

no_of_rated_products_per_user.describe()

quantiles = no_of_rated_products_per_user.quantile(np.arange(0,1.01,0.01), interpolation='higher')

plt.figure(figsize=(10,10))
plt.title("Quantiles and their Values")
quantiles.plot()
# quantiles with 0.05 difference
plt.scatter(x=quantiles.index[::5], y=quantiles.values[::5], c='orange', label="quantiles with 0.05 intervals")
# quantiles with 0.25 difference
plt.scatter(x=quantiles.index[::25], y=quantiles.values[::25], c='m', label = "quantiles with 0.25 intervals")
plt.ylabel('No of ratings by user')
plt.xlabel('Value at the quantile')
plt.legend(loc='best')
plt.show()

print('\n No of rated product more than 50 per user : {}\n'.format(sum(no_of_rated_products_per_user >= 50)) )

7: Popularity-Based Recommendation

#Getting the new dataframe which contains users who has given 50 or more ratings

new_df=electronics_data.groupby("productId").filter(lambda x:x['Rating'].count() >=50)

no_of_ratings_per_product = new_df.groupby(by='productId')['Rating'].count().sort_values(ascending=False)

fig = plt.figure(figsize=plt.figaspect(.5))
ax = plt.gca()
plt.plot(no_of_ratings_per_product.values)
plt.title('# RATINGS per Product')
plt.xlabel('Product')
plt.ylabel('No of ratings per product')
ax.set_xticklabels([])

plt.show()

#Average rating of the product 

new_df.groupby('productId')['Rating'].mean().head()

new_df.groupby('productId')['Rating'].mean().sort_values(ascending=False).head()

#Total no of rating for product

new_df.groupby('productId')['Rating'].count().sort_values(ascending=False).head()

ratings_mean_count = pd.DataFrame(new_df.groupby('productId')['Rating'].mean())

ratings_mean_count['rating_counts'] = pd.DataFrame(new_df.groupby('productId')['Rating'].count())

ratings_mean_count.head()

ratings_mean_count['rating_counts'].max()

plt.figure(figsize=(8,6))
plt.rcParams['patch.force_edgecolor'] = True
ratings_mean_count['rating_counts'].hist(bins=50)

plt.figure(figsize=(8,6))
plt.rcParams['patch.force_edgecolor'] = True
ratings_mean_count['Rating'].hist(bins=50)

plt.figure(figsize=(8,6))
plt.rcParams['patch.force_edgecolor'] = True
sns.jointplot(x='Rating', y='rating_counts', data=ratings_mean_count, alpha=0.4)

popular_products = pd.DataFrame(new_df.groupby('productId')['Rating'].count())
most_popular = popular_products.sort_values('Rating', ascending=False)
most_popular.head(30).plot(kind = "bar")

8: Collaborative filtering (Item-Item recommendation)

from surprise import KNNWithMeans
from surprise import Dataset
from surprise import accuracy
from surprise import Reader
import os
from surprise.model_selection import train_test_split

#Reading the dataset
reader = Reader(rating_scale=(1, 5))
data = Dataset.load_from_df(new_df,reader)

#Splitting the dataset
trainset, testset = train_test_split(data, test_size=0.3,random_state=10)

# Use user_based true/false to switch between user-based or item-based collaborative filtering
algo = KNNWithMeans(k=5, sim_options={'name': 'pearson_baseline', 'user_based': False})
algo.fit(trainset)

# run the trained model against the testset
test_pred = algo.test(testset)

test_pred

# get RMSE
print("Item-based Model : Test Set")
accuracy.rmse(test_pred, verbose=True)

9: Model-based collaborative filtering system

new_df1=new_df.head(10000)
ratings_matrix = new_df1.pivot_table(values='Rating', index='userId', columns='productId', fill_value=0)
ratings_matrix.head()

ratings_matrix.shape

X = ratings_matrix.T
X.head()

X.shape

X1 = X

#Decomposing the Matrix
from sklearn.decomposition import TruncatedSVD
SVD = TruncatedSVD(n_components=10)
decomposed_matrix = SVD.fit_transform(X)
decomposed_matrix.shape

#Correlation Matrix

correlation_matrix = np.corrcoef(decomposed_matrix)
correlation_matrix.shape

X.index[75]

i = "B00000K135"

product_names = list(X.index)
product_ID = product_names.index(i)
product_ID

correlation_product_ID = correlation_matrix[product_ID]
correlation_product_ID.shape

Recommend = list(X.index[correlation_product_ID > 0.65])

# Removes the item already bought by the customer
Recommend.remove(i) 

Recommend[0:24]

Conclusion

In this project, we embarked on a comprehensive journey to build an advanced recommender system using Amazon reviews. We began by understanding the significance of personalized recommendations in enhancing user experiences and driving engagement. Leveraging the vast amount of data available in Amazon reviews, we explored different techniques and methodologies to create a powerful recommendation engine that caters to individual user preferences.

We covered various aspects, starting from data acquisition and preprocessing, where we discussed methods such as utilizing APIs, web scraping, and accessing public datasets. The acquired data was then subjected to preprocessing steps like data cleaning, noise handling, duplicate detection, and text normalization to ensure high-quality input for the recommender system.

Feature extraction and representation played a crucial role in capturing the essence of Amazon reviews. Text representation techniques like bag-of-words, TF-IDF, and n-grams were employed to convert textual data into numerical feature vectors. Additionally, we explored the incorporation of metadata, including product categories, ratings, and helpful votes, to enrich the feature representation and enhance recommendation accuracy.

Building recommender models was a pivotal stage in the project. We delved into collaborative filtering models, including user-based and item-based approaches, as well as matrix factorization techniques like SVD and NMF. Content-based filtering models utilizing techniques like cosine similarity, TF-IDF weighting, and term frequency-inverse document frequency (TF-IDF) were also explored. Hybrid recommender models that combined collaborative filtering and content-based filtering techniques were discussed, including weighted hybrid models, cascading approaches, and ensemble methods.