Skip to content

AGENTS.md

Latest commit

 

History

History
243 lines (187 loc) · 8.96 KB

AGENTS.md

File metadata and controls

243 lines (187 loc) · 8.96 KB
Raw

AGENT.md — Code Structure Narrator

IntelliJ IDEA plugin for accessible code navigation — Windows + NVDA configuration

Project Summary

Code Structure Narrator is an IntelliJ IDEA plugin that extracts code structure via PSI (Program Structure Interface) and converts it into natural language, delivered directly to the active screen reader.

  • Tier 1 (automatic): Local PSI templates narrate code structure on every caret movement — no network call, no latency.
  • Tier 2 (on-demand): Pressing the "Explain Current Context" shortcut triggers an OpenAI Responses API call for a richer, natural-language sentence.

Target platform for this agent context: Windows + NVDA

Platform Target

Setting Value
OS Windows
Screen reader NVDA (NonVisual Desktop Access)
Primary IDE target IntelliJ IDEA Community / Ultimate
Accessibility API AccessibleAnnouncerUtil.announce()

macOS + VoiceOver is out of scope for this agent context.

Architecture Overview

Caret Event
    │
    ▼
PSI Context Builder
    │
    ├──[auto]────► Tier 1 Template Engine ──► AccessibleAnnouncerUtil ──► NVDA
    │
    └──[shortcut]─► Prompt Builder ──► OpenAI Responses API
                                              │
                                    Narration Formatter ──► AccessibleAnnouncerUtil ──► NVDA
                                              │
                                    [on failure] Tier 1 template fallback

Modules

A. PSI Context Builder

  • Locates the PsiElement at the cursor position
  • Walks the PSI tree to collect enclosing class, method, and block type
  • Produces a compact structured object consumed by both tiers

Example context object:

{
  "language": "Java",
  "className": "PaymentService",
  "methodName": "processPayment",
  "nodeType": "if_statement",
  "semanticHint": "null check",
  "verbosity": "brief"
}

B. Tier 1 Template Engine

  • Maps PSI node types → short English labels via a local rule table
  • No network call. Runs on every caret move after debounce.
  • Debounce: 300–500 ms (configurable in settings)
  • Suppresses repeated announcements for identical context

Covered node types (MVP):

Code Output
if (x == null) "null check condition"
for (int i = 0; i < n; i++) "integer range loop"
public void processPayment() "method: processPayment"
retry(3, Duration.ofSeconds(1)) "method call: retry"
class PaymentService "class: PaymentService"

C. Prompt Builder

  • Converts the structured context object into an OpenAI Responses API prompt
  • Only invoked on explicit shortcut trigger
  • Does not send raw source code

D. OpenAI Client Layer

  • Calls the Responses API, handles timeout and retry
  • Returns a single complete sentence
  • Falls back to the Tier 1 template string on any failure
  • API key read via OpenAIOkHttpClient.fromEnv() — never hardcoded

E. Narration Formatter

  • Applies length limits and screen-reader-appropriate formatting
  • Passes final string to AccessibleAnnouncerUtil

Tech Stack

Component Technology
Language Kotlin
JDK Java 17 (minimum)
Build Gradle 8.13+
Plugin build plugin IntelliJ Platform Gradle Plugin 2.x
Code analysis PSI (Program Structure Interface)
Accessibility output AccessibleAnnouncerUtil.announce()
AI API OpenAI Responses API
AI SDK com.openai:openai-java
Auth OPENAI_API_KEY env variable

Plugin Configuration (plugin.xml)

Field Value
Plugin ID com.example.code-structure-narrator
Display name Code Structure Narrator
since-build 233 (IntelliJ 2023.3)
until-build Not set
depends com.intellij.modules.platform
Listeners EditorFactoryListener, AnActionListener
Extensions applicationService, projectService, toolWindow

Events & Listeners

Listener Trigger Purpose
EditorCaretListener Cursor moves Tier 1: local template narration
EditorFactoryListener New editor opens Initialise narrator
AnActionListener "Explain Context" shortcut fires Tier 2: OpenAI API call

Settings UI

Setting Options / Default Description
Tier 1 auto narration on / off (default: on) Local template on caret move
Narration verbosity brief / detailed Controls label length
Debounce interval 300 / 500 / 1000 ms Minimum time between announcements
Announce syntax role on / off (default: on) Include structural role label
OpenAI API key source Env var / IDE secure storage Never stored as plain text

API Key Handling

Hackathon setup

  • Key stored in environment variable
  • Read at runtime via OpenAIOkHttpClient.fromEnv()
  • Never hardcoded. Never committed to version control.
  • Note in demo: production version routes through a backend proxy

Hackathon vs Production

Hackathon Production
API call path Plugin → OpenAI directly Plugin → Backend proxy → OpenAI
Key location Environment variable Server-side only
Key exposure risk Present (client-side) None
Rate limit control Per key Centrally controlled

MVP Scope

Feature MVP Notes
Java file support ✅ Kotlin is Phase 2
PSI context extraction ✅ 5 structure types
Tier 1 auto narration (local templates) ✅ On every caret move, debounced
Tier 2 AI explanation (shortcut) ✅ OpenAI Responses API, on-demand only
AccessibleAnnouncerUtil integration ✅
Fallback to template on API failure ✅ Plugin must never go silent
Settings page ✅ Verbosity, Tier 1 on/off, shortcut config
Windows + NVDA test ✅ Primary test platform for this agent
Streaming responses ❌ Removed — breaks screen reader delivery
Change narration / diff ❌ Removed — accuracy too low for MVP
Terminal narration ❌ Removed — experimental API
macOS + VoiceOver ❌ Out of scope
Kotlin file support ❌ Phase 2
Backend proxy for API key ❌ Production phase

Test Scenarios (Windows + NVDA)

Scenario Expected NVDA output
Move cursor to class declaration "class: PaymentService"
Move into if block "null check condition — method: processPayment"
Enter a for loop "integer range loop"
Land on a method call "method call: retry"
Press Explain Context shortcut Full natural-language sentence from OpenAI
API unavailable / timeout Falls back to local template — no crash, no silence
Caret moves rapidly (10 moves / 200 ms) Only one announcement fires (debounce working)

NVDA-specific notes

  • Test NVDA speech output with AccessibleAnnouncerUtil.announce() early — behaviour can differ from expectations in code
  • Verify NVDA does not double-read IDE UI elements alongside plugin announcements
  • Adjust debounce interval if NVDA cuts off announcements on rapid navigation
  • Leave time to tune announcement wording and timing for NVDA's speech rhythm

Development Schedule

Phase Work Milestone
0 — Setup Gradle + plugin.xml + IntelliJ Platform 2.x Plugin loads in IDE
1 — PSI Context Builder Traverse PSI tree, identify 5 node types, produce context object Context object logged to console
2 — Tier 1 Template Engine Local template map, debounce, AnnouncerUtil wiring Caret move triggers NVDA announcement
3 — Tier 2 AI Client Prompt builder, OpenAI call, timeout, fallback Shortcut triggers AI response via NVDA
4 — Settings UI On/off, verbosity, debounce interval, key source Settings persist correctly
5 — Integration Testing NVDA validation, edge cases, fallback paths Demo ready

Team

Role Responsibilities
IntelliJ / Kotlin developer PSI traversal, caret listener, template engine, OpenAI client layer, plugin.xml, Gradle
Accessibility / UX Template wording, verbosity rules, prompt design, NVDA testing, test scenarios
Demo / Presentation (optional) Demo script, presentation materials

Key Implementation Decisions

  • Shortcut-only for AI: Calling the API on every caret move causes latency (300 ms–2 s), cost overruns, and rate-limit failures. Shortcut-only is reliable and demo-safe.
  • Fallback first: Implement the Tier 1 fallback before testing the API. Silence is worse than a simple template string.
  • Template quality: Tier 1 labels need to sound natural enough that the plugin is useful without ever triggering Tier 2.
  • API key: Use an environment variable. Do not hardcode or commit. State in the presentation that production would use a backend proxy.
  • NVDA testing: Test on the actual target platform early. Leave time to adjust timing and wording.