EE660 Mini Project: Analysis of Optimization Algorithms in Deep Learning

Overview

This project investigates the performance of various optimization algorithms—specifically SGD, Adam, and L-BFGS—on neural network. The primary goal is to compare convergence speed, generalization accuracy, and stability under different training configurations.

Repository Structure

• CNN/: Convolutional Neural Network implementations on MNIST comparing Adam and SGD.
• With Dropout/: MLP implementations on MNIST with dropout layers enabled, comparing Adam, L-BFGS, and SGD.
• Dropout Disabled/: MLP implementations on MNIST with dropout disabled. This is particularly useful for analyzing L-BFGS, which often performs better or requires deterministic gradients without the noise introduced by dropout.
• others/: Additional experiments extending the analysis to FashionMNIST and CIFAR-10 datasets.
• data/: Directory for dataset storage (MNIST, etc.).
• Figures/: Stores generated plots of training curves and confusion matrices.

Experiments

1. Optimizer Comparison: Evaluating the convergence rates and final accuracy of SGD (with momentum), Adam, and L-BFGS.
2. Architecture Analysis: Comparing fully connected networks (MLP) vs. Convolutional networks (CNN) on the MNIST task.

Requirements

The project requires a Python environment with the following libraries:

• PyTorch
• torchvision
• NumPy
• Matplotlib & Seaborn
• scikit-learn

Usage

1. Clone the repository.
2. Ensure dependencies are installed.
3. Open any notebook (The results presented in the report is implemented under the Dropout Disabled/ directory) in VS Code or Jupyter Lab.
4. Run the cells to train the model and generate performance plots.