A local-first AI desktop control layer for chat, voice, tools, automation, and runtime diagnostics.
⭐ If Shell helps you, star the repo — it directly decides what gets built next. New here? Grab a good first issue and make your first contribution today.
Version: 1.0.15 · Creator: mdshoebking · License: Apache-2.0 · Primary OS: Windows 10/11
Watch the current 16:9 Shell Web UI demo · Actual current UI screenshots
Shell AI OS Controller is a Python desktop assistant and automation platform that connects a React/Vite/WebGL interface running in Electron with AI providers, voice, local tools, desktop automation, Telegram, email, browser control, memory, RAG, telemetry, and structured runtime diagnostics.
The current repo also includes ShellAI Core, a safe opt-in AI OS controller
backend with a CLI, model router, SQLite memory, reusable skills, tool
registry, policy/audit layer, trace monitor, optimizer suggestions, manual
cron jobs, and a minimal daemon queue. The desktop app keeps the classic
backend by default and can route through ShellAI Core with
SHELLAI_BACKEND_MODE=shellai_core.
It is not an operating system replacement, not a custom AGI model, and not a self-aware system. It is a practical AI-native desktop control layer designed to make local workflows easier to run, observe, and improve.
Shell is positioned as an AI-native desktop control layer: part AI assistant, part automation platform, part local workflow console.
It is built for users who want an AI workspace that can explain readiness, route tools safely, recover from missing dependencies, and keep the human user in control.
Most AI assistants stop at chat. Shell is built around the idea that an assistant should also understand tools, runtime state, settings, voice, desktop actions, and recovery paths.
The goal is simple:
- Fast conversation.
- Real tool execution.
- Clear errors.
- Safe automation.
- Beginner-friendly install.
- Open-source growth.
Shell is designed around visible confidence:
- A first-launch welcome tour explains chat, voice, tools, and help.
- Health checks show what is ready, missing, or Windows-only.
- Text chat stays text-first unless the user explicitly asks to hear audio.
- Risky automation remains gated by safety settings.
- Release packages exclude secrets, venvs, logs, generated builds, and cloned third-party repos.
| Area | What Shell Provides |
|---|---|
| Chat | Text chat with streaming-style UI, tool routing, grounded responses, and packaged offline LLM fallback |
| Voice | Gemini voice path plus packaged offline chat/TTS fallback and low-latency voice UI |
| Voice Pipeline | Optional wake-word, Silero VAD, and local sherpa-onnx STT fallback with safe button-mode fallback |
| Tools | 460+ catalogued Python tools behind a guarded execution gateway |
| Desktop | App/window control, screenshots, clipboard, keyboard/mouse automation |
| Windows Control | Optional pywinauto UI Automation driver with PyAutoGUI/pywin32 fallback |
| Browser | Browser automation wrappers with safety gates and dry-run support |
| Telegram | Remote-control bot with Settings > API Keys controls for token, allowlist, status, start/stop, and test send |
| SMTP sending with clear Gmail app-password diagnostics | |
| Media | Image generation, QR tools, PDF tools, YouTube summaries, OCR hooks |
| Runtime | Health checks, readiness states, logs, production release gates |
| Telemetry | Browser-rendered CPU/RAM/GPU/network charts with Python runtime metrics |
| Installer | One-click Windows bootstrap plus macOS/Linux launch helpers |
| ShellAI Core | python -m shellai CLI, agent loop, model routing, memory, skills, tools, monitor, cron, daemon |
| AI OS Fabric | Coordinator/Shell/Safety/Memory/UI/Optimizer agents wired in-process behind stable APIs |
| Memory v2 | Optional local SQLite memory with tags, importance scoring, time decay, redaction, recall audit, and legacy JSON migration |
| Project RAG v2 | Optional incremental codebase index with .gitignore-style scanning, BM25/TF-IDF fallback, semantic embeddings, and coding context queries |
| Secure Sandbox | Optional per-run coding workspace with timeout enforcement, secret-scrubbed environment, audit log, rollback cleanup, and network import guard |
| Workflow Checkpoints | Optional agent workflow persistence with last-action tracking, SQLite/JSON storage, resume loading, and auditable rollback checkpoints |
| Safety | SAFE/ASK/BLOCK shell policy, dry-run behavior, audit logs, blocked destructive commands |
| Web UI | React/Vite/WebGL Shell Neural OS renderer hosted by Electron with emerald glass panels, live transcript/chart rail, central particle orb, settings, gallery, tools, and telemetry cards |
| Shell Neural Features | Streaming voice state, permanent core memory, deep focus sessions, remote access records, project folder scanning, coding context packs, and background process inspection |
Latest verified state for this repository:
mainis synced to GitHub and the latest CI/Security runs are green.- GitHub Actions test matrix passes on Python 3.10, 3.11, 3.12, and 3.13.
- Local CI-style regression passes:
538 passed. - Release integrity, public package checks, secret pattern guard, dependency audit, and CodeQL pass.
- Real Web UI probes cover Dashboard chart/chat, transcript memory, Settings scroll/API keys, Telegram panel, Gallery render/save, Control Center execution, fake camera/screen streams, animations, and voice button paths.
- Tool/agent probes scan 468 catalog entries with 0 probe errors, and 37/37 agents pass readiness/execution smoke checks.
Details:
Shell's primary desktop interface now uses a React/Vite renderer hosted by
Electron. The renderer lives in shell_web_ui/ and keeps Shell's Python
backend, tool gateway, voice pipeline, memory, RAG, and safety model behind an
Electron IPC bridge backed by shell_electron_bridge.py and the pure-Python
ShellBackendBridge.
The web renderer is the default path:
.codex_ui_venv/bin/python launch.pyFor renderer development:
cd shell_web_ui
npm install
npm run dev
SHELL_ELECTRON_DEV_URL=http://127.0.0.1:5173 ../.codex_ui_venv/bin/python ../launch.pyFor packaged/local launch without the dev server, build once:
cd shell_web_ui
npm run buildThe JavaScript bridge exposes window.shellAPI.startVoice(),
window.shellAPI.stopVoice(), window.shellAPI.executeCommand(cmd),
window.shellAPI.getSystemMetrics(), and window.shellAPI.searchMemory(query).
The renderer also polyfills the original Electron IPC calls so imported visual
components can call Shell's Python backend without changing their UI timing,
animation, or layout code.
Current Web UI notes:
- The Dashboard transcript and chart composer are both text-capable. Typed messages stay text-only; voice output is only triggered by voice-source replies or the explicit speaker control.
- Settings keeps its tab strip visible while General/API/Security content scrolls inside the panel.
- Settings > API Keys includes Telegram Remote Control setup: BotFather token, allowed chat IDs, PC-control gate, terminal gate, bot status, start/stop, and test-message send.
- Camera and screen share use browser/WebEngine media APIs, with an explicit
source selector and
STOP CAPTUREcontrol. - The checked-in real WebEngine probe is
tools/real_web_ui_cdp_probe.mjs; it exercises main tabs plus nested controls such as Settings scroll, Telegram status, Control Center tool execution, Phone error handling, Notes save, chart command routing, transcript text, fake camera/screen streams, and voice start/stop buttons.
The feature modules are documented in
docs/SHELL_NEURAL_INTEGRATION_REPORT.md, and
latency notes are tracked in
docs/SHELL_PERFORMANCE_BENCHMARK.md.
- Live telemetry charts are rendered in the React/Electron UI from Python runtime metrics.
SHELL_WAKE_WORD_ENABLED=0(default) keeps hands-free wake detection off until explicitly tested. Enable withSHELL_WAKE_WORD_ENABLED=1and provide a custom "Hey Shell" openWakeWord model viaSHELL_WAKE_WORD_MODEL_PATHS.SHELL_VAD_ENABLED=0(default) keeps the current timing endpointing path. Enable withSHELL_VAD_ENABLED=1to use Silero streaming VAD; if loading fails, Shell falls back to the existing timing logic.- Wake-word sensitivity can be tuned in Settings under Voice & Speech, or with
SHELL_WAKE_WORD_SENSITIVITY/SHELL_WAKE_WORD_THRESHOLD. SHELL_PYWINAUTO_ENABLED=0(default) keeps legacy Windows automation active. SetSHELL_PYWINAUTO_ENABLED=1on Windows to prefer pywinauto's UI Automation backend for app launch, focus, close, resize, minimize, maximize, and window listing. PyAutoGUI/pywin32 remain fallback paths.SHELL_MEMORY_V2_ENABLED=0(default) keeps the legacy JSON memory tools as the primary path. SetSHELL_MEMORY_V2_ENABLED=1to route memory tools to the local SQLite Memory v2 store. OptionalSHELL_MEMORY_V2_PATHselects the database path, andSHELL_MEMORY_V2_DECAY_DAYStunes time-decay half-life. Public APIs aresave_memory(),recall_memory(), andforget_memory()inshell_memory_v2.py;memory_v2_migrate_legacy_toolimports~/.shell_smart_memory.json.- Local STT auto-enables when the bundled sherpa-onnx streaming model exists at
models/stt/sherpa-onnx/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17/. In that state Shell uses local STT first for backend voice input and falls back to Google/SpeechRecognition only if local recognition fails. SetSHELL_LOCAL_STT_ENABLED=0to explicitly disable local STT, or setSHELL_LOCAL_STT_MODEL_DIR=/path/to/sherpa-modelto override the bundled model. The default bundled model is English-focused for low-latency English/Hinglish commands. For Hindi or multilingual offline recognition, pointSHELL_LOCAL_STT_MODEL_DIRat a sherpa-onnx Whisper export, setSHELL_LOCAL_STT_MODEL_KIND=whisper, and setSHELL_LOCAL_STT_LANGUAGE=hior another Whisper language code. SHELL_OFFLINE_LLM=1enables the local offline chat brain when the user has installed a GGUF model from Settings > General > Offline Brain. Windows setup builds do not bundle chat GGUF files by default; they bundle the runtime and expose small on-demand model choices for low-memory PCs. Shell usesllama-cpp-pythonfor typed chat, chart text, and voice-originated replies when cloud providers are unavailable. If no model/runtime is ready, Settings reportsFALLBACKand Shell keeps deterministic local answers.- Windows launchers enable
SHELL_WINDOWS_PERFORMANCE_MODE=balancedby default. This caps the bundled offline LLM context/batch/token defaults and limits BLAS worker threads so entry-level Windows PCs stay responsive. Power users can overrideSHELL_OFFLINE_LLM_CONTEXT,SHELL_OFFLINE_LLM_BATCH,SHELL_OFFLINE_LLM_MAX_TOKENS, or setSHELL_WINDOWS_PERFORMANCE_MODE=offbefore launch. SHELL_CHAT_PROVIDER_MODE=auto(default) uses a configured cloud provider first only when a non-placeholder chat API key is present and a quick online probe succeeds. SetSHELL_CHAT_PROVIDER_MODE=offlineto force the packaged offline LLM for typed chat, chart text, and voice-originated turns. Relevant local memory is included by default withSHELL_CHAT_MEMORY_CONTEXT=1, and Project RAG context is included whenSHELL_PROJECT_RAG_ENABLED=1.SHELL_PROJECT_RAG_ENABLED=0(default) keeps project indexing off. SetSHELL_PROJECT_RAG_ENABLED=1to enable incremental local codebase indexing andproject_rag_query_tool/project_rag_index_tool. OptionalSHELL_PROJECT_RAG_EMBEDDINGS_ENABLED=1enables sentence-transformers embeddings when a local/available model is configured; lexical BM25/TF-IDF fallback remains available without embeddings.SHELL_SECURE_SANDBOX_ENABLED=0(default) keeps existing direct code execution behavior. SetSHELL_SECURE_SANDBOX_ENABLED=1to route Python code execution through an isolated temporary workspace with secret-scrubbed environment variables, timeout enforcement, rollback cleanup, and JSONL audit records. Optional settings includeSHELL_SECURE_SANDBOX_TIMEOUT_S,SHELL_SECURE_SANDBOX_NETWORK,SHELL_SECURE_SANDBOX_AUDIT,SHELL_SECURE_SANDBOX_ROOT, andSHELL_SECURE_SANDBOX_KEEP_SUCCESS. Network blocking currently uses a Python import guard by default; future Docker/bubblewrap isolation can be added behind the same flag.SHELL_WORKFLOW_CHECKPOINTS_ENABLED=0(default) keeps agent workflow state persistence off. SetSHELL_WORKFLOW_CHECKPOINTS_ENABLED=1to enablesave_checkpoint(),load_checkpoint(), androllback()inshell_workflow_checkpoints.py, plus checkpoint tools for multi-step agents. Optional settings includeSHELL_WORKFLOW_CHECKPOINTS_BACKEND=sqlite|json,SHELL_WORKFLOW_CHECKPOINTS_PATH, andSHELL_WORKFLOW_CHECKPOINTS_MAX_PER_WORKFLOW.SHELL_ELECTRON_DEV_URLpoints Electron at a running Vite dev server instead ofshell_web_ui/dist/index.html.VITE_SHELL_WEB_USE_GEMINI=1re-enables the renderer's direct Gemini live voice path during web UI development. By default, the power/mic controls call Shell's Python bridge.
Current public screenshots are stored in screenshots/current/. They are real
1440x900 PNG captures from the running Shell Web UI, not handmade mockups.
These screenshots are the primary visuals for README, docs, and the landscape
Remotion demo.
| Dashboard | Control Center |
|---|---|
![]() |
![]() |
| Gallery | Settings |
|---|---|
![]() |
![]() |
| Apps | Notes |
|---|---|
![]() |
![]() |
| Phone | Macros |
|---|---|
![]() |
![]() |
Watch the current 16:9 Shell Web UI demo
| Media | Preview |
|---|---|
| Current 16:9 Web UI Demo | ![]() |
| Actual Current Dashboard | ![]() |
Recommended launch media:
- 36-second current 16:9 English Web UI demo.
- 90-second real voice demo after Gemini/remote audio setup.
- 2-minute "chat opens apps and runs tools" demo.
- 5-minute technical architecture walkthrough.
Current media files:
High-level flow:
User
-> React Web UI in Electron / Voice / Telegram
-> Electron IPC + Shell Hub + Runtime State
-> NL Router + Tool Gateway + Agent Orchestrator
-> Local Tools / APIs / Desktop Automation / Browser Automation
-> Structured Result + Logs + UI Event Stream
Optional ShellAI Core path:
User / CLI / Desktop feature flag
-> ShellAI API
-> AgentRuntime + CoordinatorAgent
-> MemoryAgent + ModelRouter + SafetyAgent
-> ToolRegistry / ShellTool / FileTool / OSTool
-> Trace + SQLite memory + UI summary
.
├── agent.py # Main AI agent runtime
├── shellai/ # ShellAI Core CLI, agent loop, fabric, memory, skills, tools
├── shell_web_ui/ # React/Vite/WebGL renderer and Electron desktop shell
├── shell_ui/ # Retired legacy desktop UI assets kept for historical compatibility
├── core/shellai_bridge.py # Feature-flagged desktop bridge to ShellAI Core
├── shell_tool_gateway.py # Tool execution gateway
├── shell_telegram.py # Telegram bot integration
├── shell_windows_mcp.py # CursorTouch Windows-MCP bridge
├── core/ # Modular runtime, health, memory, orchestration
├── installer/ # Bootstrap, health, and repair logic
├── tools/ # Release, probes, diagnostics, packaging
├── docs/ # Architecture and rollout documents
├── assets/brand/ # Official Shell logo and brand rules
├── screenshots/ # Current public UI captures
├── gifs/ # Reserved for future current UI GIFs
├── videos/ # Current 16:9 demo and Remotion source
├── banners/ # Public banner and social assets
├── .github/ # Issue and pull request templates
├── LICENSE # Apache-2.0 license
├── NOTICE # Attribution notice
├── LEGAL.md # Beginner-friendly legal report
├── SECURITY.md # Security policy
└── THIRD_PARTY_NOTICES.md # Dependency/license audit notes
For normal users:
1. Download Shell AI Windows setup EXE from the latest release.
2. Double-click shell-ai-os-controller-setup-[version].exe.
3. Launch Shell AI from the Start Menu or desktop shortcut.
The setup EXE installs Shell into your user profile, creates Start Menu
shortcuts, can optionally add a desktop shortcut, and can optionally start
Shell when Windows starts. The normal Windows app shortcut launches the bundled
ShellAI.exe; users do not need Python, Node.js, npm, or a virtual environment
installed just to run the app.
Source zip fallback:
1. Download the release zip.
2. Extract it.
3. Double-click ONE_CLICK_INSTALL.bat.
4. Double-click Start_ShellAI.bat.
What the bootstrap does:
- Detects Python.
- Creates a virtual environment.
- Installs Python dependencies.
- Installs and builds the React Shell Web UI in
shell_web_ui/. - Runs health checks.
- Prepares runtime folders.
- Starts Shell through the production launcher.
If something breaks:
Double-click Repair_ShellAI.bat
Build the Windows setup EXE on a Windows build machine:
Double-click Build_Windows_EXE.bat
The generated installer is written to dist\shell-ai-os-controller-setup-[version].exe.
The installer is a proper Inno Setup desktop app package for the bundled
offline-model release: it builds the React renderer, bundles ShellAI.exe with
PyInstaller under ShellAIApp\, and points Start Menu/Desktop/startup
shortcuts at that app executable. The batch launchers stay in the install folder
only as explicit source-mode repair/fallback tools.
The Settings > System tab can check the release feed and show UPDATE NOW
when a newer setup EXE is attached to the latest GitHub release.
The React Shell Web UI build requires Node.js/npm 20.19+ or 22.12+. On Windows,
the source bootstrap and repair flow refresh PATH after winget, resolve
npm.cmd directly, upgrade old Node LTS installs, and only then run the Web UI
install/build steps. The setup EXE already contains the built runtime for
normal users.
chmod +x ONE_CLICK_INSTALL.command start_shellai.command repair_shellai.command
./ONE_CLICK_INSTALL.command
./start_shellai.commandchmod +x start_shellai.sh repair_shellai.sh
./start_shellai.shFor a detailed beginner guide, see docs/INSTALL_BEGINNER.md.
- Documentation index
- Product experience
- Design system
- Phase 6 UI/UX audit
- Product experience design
- Screenshot and demo strategy
- Trust and credibility
- Beginner install guide
- Developer guide
- Architecture guide
- ShellAI Core and AI OS Fabric
- API and tool guide
- Troubleshooting
- FAQ
- Advanced usage
- Roadmap
- AI ecosystem roadmap
- Website plan
- Public launch plan
- Community guide
- Release process
- Enterprise architecture review
- AI infrastructure plan
- Configuration system
- Observability and debugging
- Enterprise security preparation
- Cloud infrastructure readiness
- API ecosystem
- Advanced AI orchestration
- Sync and storage strategy
- Phase 7 security infrastructure
- Plugin and automation ecosystem
- DevOps and cloud deployment
- Enterprise and product strategy
- AI agent ecosystem
- Multi-agent orchestration
- AI memory system
- Tool execution and automation
- Automation marketplace
- Agent safety and governance
- Voice and multimodal future
- Developer SDK ecosystem
- Global launch strategy
- Enterprise distribution
- Brand authority and trust
- Community growth
- Content and education
- Website and public presence
- Enterprise adoption
- Analytics and product insight
- Sustainability strategy
- Competitive positioning
- Long-term governance
- Final master ecosystem report
- Public GitHub release playbook
git clone <your-fork-url> shell-ai-os-controller
cd shell-ai-os-controller
python -m venv venv
source venv/bin/activate # macOS/Linux
# venv\Scripts\activate # Windows
pip install -r requirements.txt
cp .env.example .env
python agent.py consoleRequired for full voice mode:
GOOGLE_API_KEYLIVEKIT_API_KEYLIVEKIT_API_SECRETLIVEKIT_URL
Optional features need their own API keys or dependencies. Missing optional providers should produce clear readiness messages instead of crashing the app.
ShellAI Core CLI:
python -m shellai doctor
python -m shellai run "!pwd" --json
python -m shellai skills list
python -m shellai monitor
python -m shellai optimize
python -m shellai cron list
python -m shellai daemon statusDesktop ShellAI Core bridge is opt-in:
SHELLAI_BACKEND_MODE=shellai_core python launch.pyKeep SHELLAI_BACKEND_MODE=classic or unset for the existing desktop behavior.
python3 -m shellai doctor
python3 -m shellai run "!pwd" --json
python3 -m shellai monitor --limit 10
python3 -m shellai optimize
python3 -m shellai cron run skill_usage_report --dry-run
python3 tools/production_release_check.py --strict
python3 tools/package_public_release.py
python3 tools/production_readiness.py --run-tests
python3 tools/cloud_readiness_audit.py --fail-on-high
python3 tools/agent_ecosystem_audit.py --fail-on-high
python3 tools/launch_readiness_audit.py --fail-on-high
python3 tools/public_github_launch_audit.py --fail-on-high
python3 tools/ecosystem_master_audit.py --fail-on-high
python3 -m pytest -q| Problem | Fix |
|---|---|
| Voice is silent | Check output device, Windows volume, DISABLE_TTS, and provider/API status |
| Gemini says API key invalid | Replace GOOGLE_API_KEY with a valid key from Google AI Studio |
| Email login rejected | Use a Gmail App Password, not the normal Gmail password |
| Windows-MCP unavailable | Use Windows with Python 3.13+ and uv/uvx; macOS/Linux show a safe unsupported message |
ModuleNotFoundError |
Run the one-click installer or repair script |
| Telegram bot does not respond | Add token, enable bot, confirm allowed chat IDs and remote-control settings |
More detail: docs/INSTALL_BEGINNER.md.
Is Shell an operating system?
No. It is a desktop AI control layer that runs on top of your OS.
Can I use it commercially?
Yes. The project is Apache-2.0 licensed. Third-party APIs and services still have their own terms.
Can it control my PC from Telegram?
Yes, but only after you configure the Telegram token and enable the relevant remote-control permissions.
Does it run on macOS/Linux?
Partially. The primary target is Windows. Cross-platform UI and many tools work, but Windows-MCP is Windows-only.
Does it include API keys?
No. Users must provide their own keys in .env. Never commit .env.
- Dashboard transcript now has a small
CLEARbutton that clears persisted web UI history. - The Dashboard
CHARTbutton supports both telemetry prompts and normal text prompts:show CPU chartupdates the chart and writes a short chart reply.what is memory in Python?andexplain network protocolsroute to text chat, not telemetry.calculate 2+2routes through the Shell backend command/tool path and stays text-only.
- Text-originated chart/transcript messages do not trigger voice output.
- Settings
GENERALandAPI KEYSpanels are scrollable where needed, and Telegram Remote Control lives insideSettings > API Keys. - Real UI probes are available:
node tools/real_web_ui_cdp_probe.mjs 9235 .shell_runtime/real_web_ui_cdp_probe_loop5_finalnode tools/chart_transcript_ui_probe.mjs 9235 .shell_runtime/chart_transcript_ui_probe_loop4_clean_pass
- Add interactive approvals UI for ShellAI Core ASK-level commands.
- Add reusable checked-in visible UI probe for the ShellAI Core bridge.
- Add richer ADB, VS Code, git, and browser tool adapters to ShellAI Core.
- Add OpenSSL-backed Python runtime guidance to remove local LibreSSL urllib3 warnings.
- Improve first-run setup wizard and diagnostics UX.
- Add signed installers and macOS notarization.
- Add official documentation website.
- Add more reproducible UI screenshot and demo GIF generation.
- Harden plugin marketplace and external skill audit flow.
- Add CI release automation after Windows acceptance testing is stable.
- Add encrypted local database and sync envelope implementation.
- Publish generated OpenAPI docs after external auth is ready.
- Add durable background agent queue with supervisor watchdogs.
- Add signed automation template import/export before public marketplace.
- Complete fresh Windows acceptance test before public GA.
- Add signed Windows installer and macOS notarized app before enterprise distribution.
- Publish product website and social preview assets.
- Prepare first community-friendly good-first-issue set.
Contributions are welcome after the public repository is opened.
Start here:
Before opening a pull request:
python3 -m pytest -q
python3 tools/production_release_check.py --strictDo not commit secrets, tokens, .env, runtime logs, local chat history, or
private screenshots. See SECURITY.md.
Shell AI OS Controller is released under the Apache License 2.0
(Apache-2.0).
Copyright 2026 mdshoebking.
See:
Built by mdshoebking.
This project is being prepared as an open-source AI desktop automation platform with a focus on safety, clarity, and beginner-friendly setup.








