TouchDesigner AI Companion

A local desktop tool that captures your TouchDesigner node network via hotkey, sends the screenshot to Claude (with vision + function calling), and returns targeted, context-aware answers about your patches — with full session memory and Langfuse observability.

Stack

Layer	Tool
Language	Python 3.11+
Screenshot	mss + Pillow
Hotkey listener	pynput
MCP server	mcp Python SDK (FastMCP)
LLM	Claude Sonnet 4 (claude-sonnet-4-20250514)
Observability	Langfuse
Session storage	SQLite

How to run

# 1. Clone and enter the project
git clone <repo-url> td-companion
cd td-companion

# 2. Install dependencies
pip install -r requirements.txt

# 3. Set your API keys
cp .env.example .env
# Edit .env with your ANTHROPIC_API_KEY and Langfuse keys

# 4. Launch
python main.py

# 5. Focus your TouchDesigner window, press Ctrl+Shift+T,
#    type your question, and get an answer.

macOS: Grant Accessibility permissions to your terminal app when prompted by pynput.

How it works

1. Hotkey → Screenshot — Pressing Ctrl+Shift+T triggers mss to capture your full screen. The PNG bytes are held in memory and sent to Claude as a base64 image.
2. Claude with function calling — The image + question + full session history are sent to Claude Sonnet 4. Claude can invoke four MCP-defined tools (analyze_network, suggest_next_node, diagnose_problem, explain_node) by extracting what it sees in the screenshot and routing the question to the right analytical frame.
3. Session persistence + observability — Every turn (user question + assistant answer) is saved to a local SQLite database and logged as a Langfuse trace with model, token usage, and I/O metadata for debugging and cost tracking.

Project structure

td-companion/
├── main.py               # Entry point, hotkey listener, main loop
├── screenshot.py          # mss screen capture → PNG bytes
├── agent.py               # Claude API with vision + function calling
├── session.py             # SQLite session read/write
├── observability.py       # Langfuse trace logging
├── mcp_server/
│   ├── __init__.py
│   └── tools.py           # 4 FastMCP tools + Anthropic tool schemas
├── .env.example
├── .gitignore
└── requirements.txt