This project provides classical ML and deep-learning models for classifying IMDb reviews by negative or positive sentiment. We start with data loading and exploratory analysis, followed by text vectorization and training using several classical ML models.
The dataset used is the IMDb Large Movie Review Dataset, which contains 25,000 training and 25,000 testing reviews.
Test accuracies obtained with classical ML models are:
| Model | Accuracy (%) |
|---|---|
| Logistic Regression | ~88 |
| Random Forest | ~85 |
| Naive Bayes | ~85 |
| Support Vector Machine (SVM) | ~88 |
These results align with expected baselines on this dataset.
An LSTM model is implemented using PyTorch. With fine-tuning of parameters, the model achieves 82% accuracy on the test dataset, and begins to overfit after a few epochs. These results align with expectations for an LSTM model. The next step is to implement a different model (BERT, etc) which should be able to surpass 90% accuracy for this type of problem.
pip install -r requirements.txt pip install .Download the IMDb Large Movie Review Dataset and the GloVe embedding weights and place them in imdb_classifier/data/.
Run the Jupyter notebook classical_ml.ipynb to reproduce the classical ML workflow. Run the Jupyter notebook dl.ipynb for reproduce the deep-learning workflow.