Completed as part of the STAT483: Big Data Analytics course at the University of Bahrain.
This project applies distributed computing, Natural Language Processing (NLP), supervised machine learning, and unsupervised clustering techniques to analyze millions of Amazon product reviews using Apache Spark and PySpark.
- Ebrahim Juma Shakak Alsawan
- Sadeq Jaafar Ali Deyab
- Salman Wael Salman
- Ali Sameer Ali Alzenji
- Ahmed Sadiq Ali Alsameea
The objective of this project was to extract meaningful insights from large-scale Amazon review data using distributed machine learning and NLP techniques.
The project focused on:
- Sentiment classification
- Review clustering
- Text preprocessing
- Large-scale feature engineering
- Distributed NLP pipelines
- Pattern discovery from customer reviews
The analysis combined supervised and unsupervised machine learning workflows to evaluate customer sentiment and uncover hidden product-review structures at scale.
Amazon Reviews Dataset from Kaggle
https://www.kaggle.com/datasets/abdallahwagih/amazon-reviews
- Total Reviews: 3.8 Million
- Training Dataset: 3.6 Million Reviews
- Test Dataset: 400,000 Reviews
- Ratings 1–2 → Negative Reviews
- Ratings 4–5 → Positive Reviews
The dataset provides large-scale customer review text data suitable for distributed Natural Language Processing and machine learning analysis.
| Column | Type | Description |
|---|---|---|
reviewerID |
str | Unique identifier for the reviewer/customer |
asin |
str | Amazon Standard Identification Number (ASIN) representing the product |
reviewerName |
str | Name or username of the reviewer |
helpful |
array[int] | Helpfulness rating represented as [helpful_votes, total_votes] |
reviewText |
str | Full text review written by the customer |
overall |
int | Product rating score given by the customer (1–5 stars) |
summary |
str | Short summary/title of the review |
unixReviewTime |
int | Review timestamp stored in Unix time format |
reviewTime |
str | Human-readable review date |
| Feature | Description |
|---|---|
body_tokens |
Tokenized review text |
body_clean |
Review text after stop-word removal |
raw_features |
CountVectorizer numerical representation |
features |
TF-IDF feature vectors used for machine learning |
prediction |
Predicted sentiment class |
cluster |
Assigned K-means cluster label |
Large e-commerce platforms generate millions of customer reviews daily, making manual review analysis impossible at scale.
This project aimed to solve several analytical challenges:
- Automatically classify customer sentiment
- Identify hidden product-review groupings
- Discover review language patterns
- Build scalable NLP pipelines using distributed computing
- Analyze customer satisfaction trends efficiently
The project demonstrates how big data technologies can transform unstructured text into actionable business intelligence.
- Processed 3.8 million Amazon reviews using Apache Spark
- Leveraged distributed PySpark pipelines for scalable NLP preprocessing
- Applied Spark MLlib for machine learning at scale
- Implemented memory-efficient TF-IDF vectorization workflows
- Used parallelized transformations and distributed computations
The project implemented a complete distributed NLP preprocessing pipeline.
✔ Lowercase text normalization
✔ Regex tokenization
✔ Stop-word removal
✔ CountVectorizer transformation
✔ TF-IDF feature scaling
✔ Sparse vector generation
✔ Distributed feature engineering- Apache Spark
- PySpark
- Spark MLlib
- Logistic Regression
- K-Means Clustering
- TF-IDF Vectorization
- Bag-of-Words Modeling
- RegexTokenizer
- StopWordsRemover
- CountVectorizer
- IDF Transformation
- Pandas
- Matplotlib
- Seaborn
- WordCloud
# Lowercase normalization
train_df = train_df.withColumn("title_lower", lower(col("title")))
.withColumn("body_lower", lower(col("body")))
# Tokenization and stop-word removal
tokenizer = RegexTokenizer(inputCol="body_lower", outputCol="body_tokens")
stop_remover = StopWordsRemover(inputCol="body_tokens", outputCol="body_clean")# TF-IDF Pipeline
cv = CountVectorizer(inputCol="body_clean", outputCol="raw_features", vocabSize=5000)
idf = IDF(inputCol="raw_features", outputCol="features")
# Logistic Regression
lr = LogisticRegression(featuresCol="features", labelCol="label")
pipeline = Pipeline(stages=[cv, idf, lr])
model = pipeline.fit(train_df)# K-Means clustering
kmeans = KMeans(
k=12,
initMode="k-means||",
maxIter=20,
featuresCol="body_bow"
)
clustering_model = kmeans.fit(vectorized_data)| Metric | Score |
|---|---|
| Accuracy | 85.57% |
| Precision | 84.89% |
| Recall | 86.54% |
| F1-Score | 85.71% |
| False Positive Rate | 15.40% |
- Logistic Regression achieved strong sentiment classification performance on millions of reviews.
- Positive reviews frequently used emotional and quality-focused language.
- Negative reviews focused more heavily on product issues and transactional concerns.
- TF-IDF feature engineering significantly improved sentiment detection quality.
- Optimal clusters identified using the elbow method: K = 12
- Reviews naturally grouped by product category and vocabulary patterns.
- K-means clustering successfully separated major product themes.
| Cluster | Dominant Vocabulary |
|---|---|
| Cluster 2 | book, read, good |
| Cluster 4 | album, cd, songs, music |
| Cluster 7 | movie, film, good |
- Product-specific vocabulary strongly influenced clustering behavior.
- Review similarity naturally formed category-based groupings.
- Cluster distribution revealed both generalized and niche review categories.
Satisfied customers commonly used:
- emotional language
- evaluative descriptors
- quality-oriented vocabulary
Examples:
- great
- amazing
- perfect
- excellent
Negative reviews focused more heavily on:
- transactional concerns
- product defects
- pricing/value complaints
Examples:
- money
- bought
- problem
- return
This project successfully demonstrated:
- distributed machine learning
- scalable NLP processing
- large-scale text analytics
- memory-efficient Spark workflows
on a dataset containing millions of reviews.
.
├── data/
│ ├── train.csv
│ └── test.csv
├── notebooks/
│ ├── data_exploration.ipynb
│ ├── preprocessing.ipynb
│ ├── classification.ipynb
│ └── clustering.ipynb
├── src/
│ ├── preprocessing.py
│ ├── classification.py
│ ├── clustering.py
│ └── evaluation.py
├── images/
│ ├── confusion_matrix.png
│ ├── word_clouds.png
│ ├── cluster_distribution.png
│ └── elbow_method.png
├── requirements.txt
└── README.mdpip install pyspark pandas matplotlib seaborn wordcloud scikit-learn# Preprocessing
python src/preprocessing.py
# Classification
python src/classification.py
# Clustering
python src/clustering.py
# Evaluation
python src/evaluation.py- Deep Learning integration using LSTM/BERT
- Real-time streaming analysis using Kafka
- Multi-class sentiment classification
- Aspect-based sentiment analysis
- Advanced transformer-based NLP pipelines
- Big Data Analytics
- Distributed Computing
- Natural Language Processing (NLP)
- Machine Learning
- Sentiment Classification
- K-Means Clustering
- PySpark
- Spark MLlib
- Feature Engineering
- TF-IDF Vectorization
- Text Mining
- Scalable Data Processing
Course: STAT483 – Big Data Analytics
Institution: University of Bahrain
Project Type: Big Data NLP & Machine Learning Project
- Sadeq Jaafar Ali Deyab
- Salman Wael Salman
- Ali Sameer Ali Alzenji
- Ahmed Sadiq Ali Alsameea