This code is associated with the SparseFit: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations paper, presented at the ACL Main Conference 2024. Our work extends the research presented in Few-Shot Self-Rationalization with Natural Language Prompts. We have used their codebase as a foundation for this project.

Citation

@inproceedings{solano-etal-2024-sparsefit,
    title = "{S}parse{F}it: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations",
    author = "Solano, Jesus  and
      Sanni, Mardhiyah  and
      Camburu, Oana-Maria  and
      Minervini, Pasquale",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.113",
    pages = "2053--2077",
    }

Installation

1. Clone the repository

git clone https://github.com/SanniM3/predicitons_with_explanations.git
cd predicitons_with_explanations

2. Download and install Conda.

3. Create and activate a Conda environment.

conda create -n feb python=3.10.12
conda activate feb

4. Download the requirements.

pip install -r requirements.txt
pip install --upgrade deepspeed
python -m spacy download en_core_web_sm

wandb offline

Train/Eval Data

Preferred: Download

The following command will download the datasets (all except E-SNLI), but doesn't give splits we used for our experiments.

wget https://storage.googleapis.com/feb-data/data.zip
unzip data.zip

FEB Splits

For our experiments, we didn't save and load splits, but generated them for every run inside input_to_label_and_rationale.py. If you'd like to have the same splits that we used stored locally, you should run the following:

mkdir feb_data
python feb_benchmark.py

Replicate Data

If you want to replicate our preprocessing steps, you can do so with the following commands:

SBIC

wget https://homes.cs.washington.edu/~msap/social-bias-frames/SBIC.v2.tgz
tar -xvzf SBIC.v2.tgz
python scripts/preprocess_data.py --dataset sbic --dataset_path <path-to-sbic-folder>`

SenseMaking

python scripts/preprocess_data.py --dataset sensemaking --dataset_path <path-to-sensemaking-folder>

ECQA

git clone https://github.com/dair-iitd/ECQA-Dataset.git

cd ECQA-Dataset 
mkdir cqa
cd cqa
wget https://s3.amazonaws.com/commensenseqa/train_rand_split.jsonl 
wget https://s3.amazonaws.com/commensenseqa/dev_rand_split.jsonl

cd explain_lm/finetuning
python scripts/preprocess_data.py --dataset ecqa --dataset_path <path to `ECQA-Dataset`>

Run evaluation

T5/UnfifiedQA

The command below will train and evaluate given models on chosen datasets with 60 random seeds:

python scripts/exp.py --exp_root <path_to_checkpoints_folder> --not_dryrun --model_vals <a string of models to evaluate separated by comma> --dataset_vals <a string of datasets to evaluate on, separated by comma> --n_gpus <number of available GPUs>

By default these experiments will be done with IO formats (prompts) that find to work the best (according to the experiments in the paper), but you can play around with different values in format_dict in scripts/exp.py.

The same command with concrete values:

mkdir checkpoints
python scripts/exp.py --exp_root checkpoints --not_dryrun --model_vals t5-base,t5-large,t5-3b --dataset_vals esnli --n_gpus 4 --model_class t5
python scripts/exp.py --exp_root checkpoints --not_dryrun --model_vals allenai/unifiedqa-t5-base,allenai/unifiedqa-t5-large,allenai/unifiedqa-t5-3b --dataset_vals ecqa,sensemaking,sbic --n_gpus 4 --model_class t5

Llama-7B

python scripts/exp.py --exp_root checkpoints --not_dryrun --model_vals meta-llama/Llama-2-7b-hf --dataset_vals esnli,ecqa,sensemaking,sbic --n_gpus 4 --model_class llama

GPT3

python scripts/exp.py --exp_root <path_to_checkpoints_folder> --model_vals <a string of models to evaluate separated by comma> --dataset_vals <a string of datasets to evaluate on, separated by comma>  --use_gpt3 --openai_key <your_openai_key>  --not_dryrun

Collect results

After you're doing with training/eval with 60 seeds, you can collect results (mean, stddev) by running this:

mkdir out
python scripts/exp.py --exp_root <path_to_checkpoints_folder>  --collect_results

If you get the assertion error, check which runs have not been trained properly, repeat evaluating only those seeds, and run the above command again.

Human evaluation

We use the NLEs associated with the first 30 correctly predicted samples in each validation set in the training for human evaluation. To make the evaluation more robust, 30 samples were chosen to be balanced in the number of classes.

To get the generations from which to sample from, run:

bash get_generations.sh