Lightweight Windows UI element detector for UI-agent fallback. When native UI Automation (e.g. pywinauto, UIAutomation) fails to locate elements, this library provides a computer-vision fallback: detect interactive UI elements from a screenshot, find elements by text via OCR, compute safe click points, and verify actions via before/after screenshot comparison.
| Metric | Value |
|---|---|
| mAP@50 | 0.989 |
| mAP@50-95 | 0.954 |
| Precision | 0.996 |
| Recall | 0.973 |
Model: YOLO11s (~9.4M params, ~21.3 GFLOPs) | CPU inference: 44-79 ms (Apple M2 Pro)
- Element Detection — YOLO11s detector trained on 3 000 synthetic Windows-style UI screenshots. Detects:
button,textbox,checkbox,dropdown,icon,tab,menu_item. - Text Search — Combine OCR (EasyOCR) + detector bboxes + fuzzy matching (rapidfuzz) to find the element whose text best matches a query.
- Safe Click Points — Given a bbox, compute a point safely inset from borders.
- Action Verification — Compare before/after screenshots via pixel diff, OCR delta, or combined mode.
- Synthetic Dataset Generator — HTML/CSS templates rendered via Playwright with domain randomization (themes, fonts, scaling, noise).
pip install -e .For dataset generation (requires Playwright):
pip install -e ".[datagen]"
playwright install chromiumFor the Gradio demo:
pip install -e ".[demo]"Model weights are downloaded automatically from HuggingFace on first run.
from local_ui_locator import detect_elements, find_by_text, safe_click_point
# Detect all UI elements (weights auto-download on first run)
detections = detect_elements("screenshot.png", conf=0.3)
for det in detections:
print(f"{det.type}: {det.bbox} (score={det.score:.2f})")
# Find element by text
match = find_by_text("screenshot.png", query="Sign in")
if match:
x, y = safe_click_point(match.bbox)
print(f"Click at ({x}, {y})")Generate synthetic Windows-style UI screenshots with YOLO-format annotations:
python data_gen/generate.py --out datasets/ui_synth_v2 --n 3000 --seed 42Visualize samples:
python data_gen/visualize.py --data datasets/ui_synth_v2 --n 20Train YOLO11s on the synthetic dataset:
python training/train_yolo.py --data datasets/ui_synth_v2/data.yaml --epochs 80Evaluate and generate prediction samples:
python training/eval.py --weights weights/best.pt --data datasets/ui_synth_v2/data.yamlcd demo_space && python app.pyUpload a screenshot, adjust confidence, filter classes, and search elements by text.
├── data_gen/ # Synthetic dataset generator
│ ├── templates/ # HTML screen templates
│ ├── generate.py # Main generation script
│ ├── render.py # Playwright rendering + bbox extraction
│ ├── labels.py # YOLO label conversion
│ ├── augment.py # Image augmentation pipeline
│ └── visualize.py # Bbox visualization
├── datasets/ # Generated datasets (data.yaml configs)
├── training/ # Training & evaluation scripts
├── local_ui_locator/ # Python inference package
│ ├── detector.py # YOLO wrapper
│ ├── ocr.py # EasyOCR wrapper
│ ├── search.py # Text-based element search
│ ├── verify.py # Action verification
│ ├── schemas.py # Data models
│ └── utils.py # Utilities
├── api/ # FastAPI backend
├── web/ # Next.js frontend
├── examples/ # Usage examples
├── weights/ # Model weights (.gitignored)
├── Dockerfile
└── requirements.txt
- YOLO11s: Upgraded from YOLOv8n for better accuracy (mAP50 0.989 vs 0.93) with moderate speed trade-off (~60ms vs ~30ms on CPU). Still fast enough for UI-agent fallback.
- EasyOCR: Pure pip-installable, no system dependencies (unlike Tesseract). Supports 80+ languages.
- Synthetic data via Playwright: HTML/CSS templates produce pixel-perfect Windows-style UI with exact bbox coordinates from DOM queries — no manual annotation needed.
- Fuzzy text matching:
rapidfuzz.fuzz.token_set_ratiohandles partial matches, word reordering, and minor OCR errors robustly.
MIT — see LICENSE.