STEB: Style Text Embedding Benchmark

STEB is a framework for evaluating style text embeddings across a variety of tasks and datasets. It is modular and extensible, making it straightforward to add new models, datasets, and evaluation tasks. Read the paper here.

Leaderboard — current results for every benchmarked model under both the Operational and Definitional STEB scores. New submissions land here automatically when a contributor's PR merges; see Submitting your model below.

Installation

git clone https://github.com/rrivera1849/STEB.git
cd STEB
python3 -m venv venv
source venv/bin/activate
pip install -e .
./download_datasets.sh

The download_datasets.sh script invokes Python tools (gdown, etc.) declared in requirements.txt, so the venv must be active when running it. Use --purge to force a clean re-download.

To download to a non-default location, pass the target directory as an argument. You then need to tell STEB where to look, either via the STEB_RAW_DATASETS_DIR environment variable or via config.ini (see Configuration):

./download_datasets.sh /path/to/raw_datasets
export STEB_RAW_DATASETS_DIR=/path/to/raw_datasets

Quick Start

Run the standard STEB benchmark — the configuration reported in the paper:

steb "rrivera1849/LUAR-MUD"

Run a specific task on a specific dataset, for example, the PAN13 authorship verification benchmark:

steb pre_defined_pair_classification "rrivera1849/LUAR-MUD" \
    --dataset pan13_authorship_verification_english_test

Configuration

STEB resolves four directory paths from environment variables, a config.ini file, and built-in defaults. Resolution order is:

1. Environment variable
2. ./config.ini (in the current working directory)
3. ~/.steb/config.ini (per-user fallback)
4. Built-in default

Variable	config.ini key	Default	Description
STEB_RAW_DATASETS_DIR	raw_datasets_dir	./raw_datasets	Raw downloaded datasets
STEB_RESULTS_DIR	results_dir	./results	Evaluation results
STEB_PROCESSED_DATA_DIR	processed_dataset_dir	~/.local/share/steb/processed_datasets	Processed dataset cache

Environment variables

export STEB_RAW_DATASETS_DIR=/path/to/raw_datasets
export STEB_RESULTS_DIR=/path/to/results
export STEB_PROCESSED_DATA_DIR=/path/to/processed

config.ini

Equivalent setup via ./config.ini (project-local) or ~/.steb/config.ini (user-wide):

[Application_Paths]
raw_datasets_dir = /path/to/raw_datasets
results_dir = /path/to/results
processed_dataset_dir = /path/to/processed_datasets

Documentation

For the full reference — CLI flags, task descriptions, supported model types, dataset config schema, and guides for adding new models or datasets, see:

STEB Documentation (also browsable as Markdown under docs/)

Submitting your model

To get your model on the public leaderboard, the short version is:

./scripts/download_results.sh                                   # 1. fetch the leaderboard raw scores
STEB_RESULTS_DIR=./submitted_results steb "<org/your-model>"    # 2. run your STEB model
python -m scripts.benchmark_clustering                          # 3. regenerate STEB scores
# 4. open scores.xlsx, and see your models standing under the STEB_operational and STEB_definitional sheets
# 5. append a 4-key entry to SUBMISSIONS.yaml, add yourself to scripts/models_all.txt
python scripts/validate_submission.py                           # 5. validate, then open the PR

The SUBMISSIONS.yaml entry is:

- short_name: <your-short-name>
  hf_id: <org/your-model>
  run_command: steb "<org/your-model>"
  contributor: <your-github-handle>

Citation

If you use STEB in your research, please cite:

  @misc{rivera-soto-etal-2026-STEB,
    title         = {STEB: A Style Text Embedding Benchmark},
    author        = {Rivera Soto, Rafael A. and Wegmann, Anna and Aggazzotti, Cristina},
    year          = {2026},
    eprint        = {2606.31741},
    archivePrefix = {arXiv},
    primaryClass  = {cs.CL},
    url           = {https://arxiv.org/abs/2606.31741}
  }

License

STEB is released under the Apache License 2.0.