Real-time American Sign Language recognition with an efficient pose-based Transformer — from raw video all the way to spoken English.
SignStream turns a signed video clip into a fluent spoken sentence through a full four-stage pipeline: pose extraction → sign recognition → natural-language generation → speech. The recognition core is an 18.65 M-parameter pose-based Transformer trained on the 2,731-class ASL Citizen dataset (NeurIPS 2023). It reaches 74.70% validation and 67.40% test top-1 accuracy — competitive with, and on test ahead of, a 25 M-parameter I3D video baseline, while using ~25% fewer parameters and running on landmark features instead of raw pixels.
| Split | Top-1 | Top-5 | Samples |
|---|---|---|---|
| Validation | 74.70% | 93.23% | 10,304 |
| Test | 67.40% | 89.31% | 32,929 |
Model: 18,651,691 parameters · 2,731 classes · 1,725-dim input features · best epoch 94
Reference points
- ASL Citizen I3D baseline reported at 25 M params / 63.10% test → SignStream reaches 67.40% test at ~0.75× the parameters.
- Prior in-project V1 baseline (5.76 M params): 67.46% val → V3-Large adds +7.24% val top-1.
All numbers are reproduced bit-exactly from the training notebook under fixed seeds and deterministic CUDA settings.
Most sign-language systems stop at classifying a single sign. SignStream is built as a deployable communication tool: it recognizes the sign, converts the recognized gloss into grammatical English with a fine-tuned T5 sequence-to-sequence model, and speaks the result — the full path a real accessibility product would need. It is also designed to be efficient: pose landmarks plus a compact Transformer instead of a heavy 3D-CNN over raw video, so it can run in real time rather than only offline.
flowchart LR
A[ASL Video] --> B["MediaPipe Holistic<br/>pose + hands + face"]
B --> C["1,725-dim<br/>per-frame features"]
C --> D["SignTransformer V3-Large<br/>Temporal CNN + 8-layer encoder<br/>18.65M params"]
D --> E["Gloss<br/>(2,731 classes)"]
E --> F["Gloss → Text<br/>rules + fine-tuned T5"]
F --> G["Fluent English sentence"]
G --> H["Text-to-Speech"]
The recognition model (SignTransformer V3-Large) pairs a temporal CNN front-end (Conv1D → BN → ReLU → Conv1D → BN with residual connections) that captures local motion, with an 8-layer Transformer encoder (d_model 384, 12 heads) that models long-range temporal structure across the whole sign.
Regularization tuned for a model this size on ~83K clips:
- Stochastic Depth (DropPath, p = 0.1) — scaled linearly across the 8 layers
- EMA of weights (decay 0.999) — inference runs on the smoothed shadow
- Mixup (α = 0.3) — gentle, so it preserves motion patterns
- Supervised Contrastive loss + signer-aware augmentation — robustness across signers
- Cosine schedule with warm restarts, label smoothing, gradient clipping, fp16 mixed precision, gradient accumulation (effective batch 128)
This repo is structured as a maintainable codebase:
src/package — models, data, training, inference, NLP, and TTS cleanly separated- Two model families side by side — the V1 baseline stays frozen for reproducibility; V3-Large is opt-in via a single import
- Real-time inference engine (
src/inference/realtime_engine.py) for live webcam use - ONNX export (
scripts/export_onnx.py) for deployment - T5 fine-tuning script (
scripts/finetune_t5.py) for the gloss → text stage - Test suite (
tests/test_signstream.py, pytest) covering data, model, and training paths - Per-signer fairness analysis — accuracy broken down per signer, not just aggregate
- Reproducibility framework — deterministic algorithms, seeded runs, SHA-verified checkpoints
signstream/
├── configs/config.yaml # model_v3 / training_v3 / augmentation sections
├── src/
│ ├── models/ # sign_transformer.py (V1) + sign_transformer_v3.py (V3-Large)
│ ├── training/ # trainer, trainer_v3 (Mixup + EMA), evaluator, helpers
│ ├── data/ # dataset + MediaPipe feature extractor
│ ├── inference/ # real-time engine
│ ├── nlp/ # gloss → English (rules + T5)
│ ├── tts/ # speech synthesis
│ └── utils/ # config loading
├── scripts/ # train, export_onnx, finetune_t5, run_realtime
├── notebooks/ # feature extraction + pipeline
├── tests/ # pytest suite
├── requirements.txt
└── pyproject.toml
# 1. Install
pip install -r requirements.txt
# 2. Train V3-Large (reads configs/config.yaml)
python scripts/train.py
# 3. Real-time demo (webcam)
python scripts/run_realtime.py
# 4. Export to ONNX
python scripts/export_onnx.py
# 5. Run tests
pytestSwitching backbones is a single import — the trainer is backbone-agnostic:
from src.models import build_model_v3_large
from src.training import TrainerV3
model = build_model_v3_large(cfg, input_dim=1725, num_classes=2731)
TrainerV3(cfg, model, train_loader, val_loader, num_classes=2731).train()Every run is deterministic and checkpointed with best/last weights (plus EMA shadow), per-epoch metrics, per-signer breakdowns, and SHA256 hashes of all artifacts.
os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True, warn_only=True)PyTorch · Hugging Face Transformers (T5) · MediaPipe Holistic · ONNX · NumPy · pytest
ASL Citizen (NeurIPS 2023) — ~83K isolated-sign videos across 2,731 glosses.
Mohammad Kheir Alhomsi — AI Engineer Syrian Private University — B.Eng. Information Engineering (AI & Data Science) Research supervised by Dr. Nisreen Sulayman and Eng. Zeinab Baghdadi.
@article{alhomsi2026signstream,
title = {SignStream: An Efficient Pose-Based Transformer for Real-Time
Isolated ASL Recognition},
author = {Alhomsi, Mohammad Kheir and Sulayman, Nisreen and Baghdadi, Zeinab},
year = {2026},
note = {Prepared for submission to IEEE Access}
}Released under the MIT License — see LICENSE.