SignStream

Real-time American Sign Language recognition with an efficient pose-based Transformer — from raw video all the way to spoken English.

SignStream turns a signed video clip into a fluent spoken sentence through a full four-stage pipeline: pose extraction → sign recognition → natural-language generation → speech. The recognition core is an 18.65 M-parameter pose-based Transformer trained on the 2,731-class ASL Citizen dataset (NeurIPS 2023). It reaches 74.70% validation and 67.40% test top-1 accuracy — competitive with, and on test ahead of, a 25 M-parameter I3D video baseline, while using ~25% fewer parameters and running on landmark features instead of raw pixels.

Model: 18,651,691 parameters · 2,731 classes · 1,725-dim input features · best epoch 94

Reference points

• ASL Citizen I3D baseline reported at 25 M params / 63.10% test → SignStream reaches 67.40% test at ~0.75× the parameters.
• Prior in-project V1 baseline (5.76 M params): 67.46% val → V3-Large adds +7.24% val top-1.

All numbers are reproduced bit-exactly from the training notebook under fixed seeds and deterministic CUDA settings.

Most sign-language systems stop at classifying a single sign. SignStream is built as a deployable communication tool: it recognizes the sign, converts the recognized gloss into grammatical English with a fine-tuned T5 sequence-to-sequence model, and speaks the result — the full path a real accessibility product would need. It is also designed to be efficient: pose landmarks plus a compact Transformer instead of a heavy 3D-CNN over raw video, so it can run in real time rather than only offline.

flowchart LR
    A[ASL Video] --> B["MediaPipe Holistic<br/>pose + hands + face"]
    B --> C["1,725-dim<br/>per-frame features"]
    C --> D["SignTransformer V3-Large<br/>Temporal CNN + 8-layer encoder<br/>18.65M params"]
    D --> E["Gloss<br/>(2,731 classes)"]
    E --> F["Gloss → Text<br/>rules + fine-tuned T5"]
    F --> G["Fluent English sentence"]
    G --> H["Text-to-Speech"]

The recognition model (SignTransformer V3-Large) pairs a temporal CNN front-end (Conv1D → BN → ReLU → Conv1D → BN with residual connections) that captures local motion, with an 8-layer Transformer encoder (d_model 384, 12 heads) that models long-range temporal structure across the whole sign.

Regularization tuned for a model this size on ~83K clips:

• Stochastic Depth (DropPath, p = 0.1) — scaled linearly across the 8 layers
• EMA of weights (decay 0.999) — inference runs on the smoothed shadow
• Mixup (α = 0.3) — gentle, so it preserves motion patterns
• Supervised Contrastive loss + signer-aware augmentation — robustness across signers
• Cosine schedule with warm restarts, label smoothing, gradient clipping, fp16 mixed precision, gradient accumulation (effective batch 128)

This repo is structured as a maintainable codebase:

• src/ package — models, data, training, inference, NLP, and TTS cleanly separated
• Two model families side by side — the V1 baseline stays frozen for reproducibility; V3-Large is opt-in via a single import
• Real-time inference engine (src/inference/realtime_engine.py) for live webcam use
• ONNX export (scripts/export_onnx.py) for deployment
• T5 fine-tuning script (scripts/finetune_t5.py) for the gloss → text stage
• Test suite (tests/test_signstream.py, pytest) covering data, model, and training paths
• Per-signer fairness analysis — accuracy broken down per signer, not just aggregate
• Reproducibility framework — deterministic algorithms, seeded runs, SHA-verified checkpoints

signstream/
├── configs/config.yaml            # model_v3 / training_v3 / augmentation sections
├── src/
│   ├── models/                    # sign_transformer.py (V1) + sign_transformer_v3.py (V3-Large)
│   ├── training/                  # trainer, trainer_v3 (Mixup + EMA), evaluator, helpers
│   ├── data/                      # dataset + MediaPipe feature extractor
│   ├── inference/                 # real-time engine
│   ├── nlp/                       # gloss → English (rules + T5)
│   ├── tts/                       # speech synthesis
│   └── utils/                     # config loading
├── scripts/                       # train, export_onnx, finetune_t5, run_realtime
├── notebooks/                     # feature extraction + pipeline
├── tests/                         # pytest suite
├── requirements.txt
└── pyproject.toml

# 1. Install
pip install -r requirements.txt

# 2. Train V3-Large  (reads configs/config.yaml)
python scripts/train.py

# 3. Real-time demo (webcam)
python scripts/run_realtime.py

# 4. Export to ONNX
python scripts/export_onnx.py

# 5. Run tests
pytest

Switching backbones is a single import — the trainer is backbone-agnostic:

from src.models import build_model_v3_large
from src.training import TrainerV3

model = build_model_v3_large(cfg, input_dim=1725, num_classes=2731)
TrainerV3(cfg, model, train_loader, val_loader, num_classes=2731).train()

Every run is deterministic and checkpointed with best/last weights (plus EMA shadow), per-epoch metrics, per-signer breakdowns, and SHA256 hashes of all artifacts.

os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True, warn_only=True)

PyTorch · Hugging Face Transformers (T5) · MediaPipe Holistic · ONNX · NumPy · pytest

ASL Citizen (NeurIPS 2023) — ~83K isolated-sign videos across 2,731 glosses.

Mohammad Kheir Alhomsi — AI Engineer Syrian Private University — B.Eng. Information Engineering (AI & Data Science) Research supervised by Dr. Nisreen Sulayman and Eng. Zeinab Baghdadi.

@article{alhomsi2026signstream,
  title   = {SignStream: An Efficient Pose-Based Transformer for Real-Time
             Isolated ASL Recognition},
  author  = {Alhomsi, Mohammad Kheir and Sulayman, Nisreen and Baghdadi, Zeinab},
  year    = {2026},
  note    = {Prepared for submission to IEEE Access}
}

Released under the MIT License — see LICENSE.