Semi Workbench is a high-performance orchestration platform designed to automate the extraction and structuring of complex industrial and MRO (Maintenance, Repair, and Operations) component data.
By combining real-time web scraping via SearxNG and Crawl4AI with a resilient, multi-provider LLM cascade (NVIDIA NIM, Groq, OpenRouter, and Gemini), the platform reliably normalizes raw input data into 100+ highly structured JSON attributes.
- Multi-Provider LLM Cascade: Built-in intelligent fallback routing (
NIM→Groq→OpenRouter→Gemini) ensures maximum uptime and rate-limit evasion. - Real-Time Data Streaming: FastAPI Server-Sent Events (SSE) stream live processing updates directly to the React (Vite) frontend dashboard.
- Autonomous Web Scraping: Headless Playwright integration via Crawl4AI to bypass bot-protection and extract raw HTML for semantic parsing.
- Self-Healing Schemas: Aggressive JSON repairing algorithms instantly recover and format broken LLM outputs.
The pipeline manages the entire data lifecycle from user injection to final CSV generation.
graph TD
A[User Uploads CSV] -->|Frontend| B(FastAPI Ingest Endpoint)
B --> C{Row Processor Queue}
C --> D[1. LLM Query Generation]
D --> E[2. SearxNG Meta Search]
E --> F[3. Crawl4AI Headless Scrape]
F --> G{4. LLM Fallback Router}
G -->|Primary: 90s Timeout| H[NVIDIA NIM]
G -->|Secondary| I[Groq API Cascade]
G -->|Tertiary| J[OpenRouter Cascade]
H -.-> K[Repair JSON & Validate]
I -.-> K
J -.-> K
K --> L[5. Merge with Original Row]
L --> M((Output CSV & SSE Stream))
The extraction pipeline is highly tuned for production efficiency.
| Component | Baseline | Optimized | Optimization Rationale |
|---|---|---|---|
| LLM Inference Speed | 120s - 300s (Timeouts) | ~15s per row | Disabled reasoning blocks (enable_thinking: False) in NVIDIA NIM to bypass generation bloat and enforce instant JSON outputs. |
| Pipeline Reliability | High failure rate on 429s | 99.9% Uptime | Integrated an aggressive LLM Fallback Cascade rotating through multiple free-tier keys. |
| Scraper Execution | 0% (Missing Binaries) | 100% Success | Set PLAYWRIGHT_BROWSERS_PATH=0 to guarantee binary alignment for Crawl4AI inside the container. |
- Docker and Docker Compose
- Valid API Keys (NVIDIA NIM, Groq, OpenRouter)
- Clone the repository and configure your environment:
git clone https://github.com/VarshneysvAI/semi-workbench.git
cd semi-workbench
# Configure API Keys
echo "LLM_API_KEY_NIM=nvapi-your-key-here" > backend/.env
echo "GROQ_API_KEYS=gsk_key1,gsk_key2" >> backend/.env
echo "OPENROUTER_API_KEYS=sk-or-v1-key1,sk-or-v1-key2" >> backend/.env- Build and launch the multi-container environment:
docker-compose build --no-cache
docker-compose up -d- Access the dashboard:
Open your browser and navigate to
http://localhost:80.
Note
The Dockerfile handles a multi-stage build, compiling the Vite frontend and serving it statically through the FastAPI backend automatically.
semi-workbench/
├── backend/ # FastAPI server and core orchestrator logic
│ ├── pipeline/ # Data lifecycle pipelines, web search, scraping logic
│ ├── providers/ # LLM Engine API adapters (nim, groq, openrouter, gemini)
│ └── live_app.py # FastAPI entry point (Server-Sent Events)
├── dashboard/ # React (Vite) Frontend UI
│ ├── src/
│ │ ├── engine/ # State context and API event stream handlers
│ │ └── views/ # User interface components and dashboards
├── searxng/ # Self-hosted SearxNG Meta-Search engine configuration
├── Dockerfile # Multi-stage Docker build (Frontend + Backend + Playwright)
└── docker-compose.yml # Multi-container orchestration (Backend + SearxNG)
For autonomous agents or developers looking to modify the routing logic:
Important
The central fallback logic is defined in backend/pipeline/provider_router.py. Modify the fallbacks array to adjust the secondary engine priorities.
When rebuilding the Docker image, do not remove ENV PLAYWRIGHT_BROWSERS_PATH=0 in the Dockerfile, as it guarantees Crawl4AI can locate the headless Chromium binary within the Python environment.