feat(agents): add Grok CLI agent support - #183
Open
bsy wants to merge 7 commits into
Open
Conversation
Wire native --agent grok through config, factory, and docs so overnight runs can use xAI's grok harness with structured streaming-json output.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Add native Grok CLI agent support so gnhf can run overnight loops with xAI's grok harness
What Changed
grokagent (src/core/agents/grok.ts) wired throughfactory.ts, theAGENT_NAMESlist, reserved-arg validation, and the bootstrapped config template/README agent table. It spawns the xAIgrokCLI per iteration with-p --output-format streaming-json --json-schema, defaults to--always-approve, and readsstructuredOutputplus token usage from the finalendevent (so--max-tokensstops after an iteration rather than mid-iteration).spawnAgentProcessinstream-utils.tsis now the single spawn path for claude/codex/copilot/pi/grok and owns Windows.cmd/.batshim detection pluscmd.exeargv escaping, andparseAgentOutputJsoninjson-extract.tsreplaces the duplicated schema-preferring parse-and-validate routine.json-extractandstream-utilstests, and an e2e suite driving a mock grok CLI fixture over the real wire format; refreshed AGENTS.md, CONTRIBUTING.md,skills/gnhf/SKILL.md, and the bug-report template accordingly.Risk Assessment
.cmd-shim path that the change set out to harden still silently truncates the multi-line prompt (and hands its remainder to cmd.exe) while the README continues to advertise shim support, which is worth resolving as a follow-up rather than blocking this merge.Testing
Built the CLI, ran the targeted unit suites for the grok agent and the shared stream/JSON helpers the refactor touched (238 agent tests) plus the three grok e2e cases, then proved the intent end-to-end at the product level: the real xAI
grok0.2.114 CLI is installed and authenticated on this machine, so I ran gnhf's actual loop against it in a scratch repo and it produced a genuine code change committed on agnhf/branch (113K in / 829 out tokens reported, 1 good iteration). I also cross-checked every flag gnhf emits against the realgrok --helpso the wire format is validated against the actual harness rather than only the fixture, and manually exercised config-file agent selection with path/arg overrides, the--max-tokensguarantee that a finished iteration is committed instead of rolled back, and grok's presence in--help, the unknown-agent error, and the bootstrapped config. Evidence includes rendered screenshots of the live TUI and exit summary plus raw CLI transcripts, git logs, and debug-log events. Everything passed; no product or setup issues surfaced. The screenshots are VT replays of the captured terminal streams rendered through headless Chrome rather than photos of a live terminal, so the ASCII banner's box-drawing glyphs sit slightly apart in the browser font -- a replay artifact, not a product defect./var/folders/92/rcfm4q4s51bcz30v1tw_vjl00000gn/T/no-mistakes-evidence/01KYSTS1VAGJKQVWGK8Z5HSVE6/gnhf-grok-REAL-live-tui.png)/var/folders/92/rcfm4q4s51bcz30v1tw_vjl00000gn/T/no-mistakes-evidence/01KYSTS1VAGJKQVWGK8Z5HSVE6/gnhf-grok-REAL-exit-summary.png)Evidence: Real grok CLI run: commit and diff gnhf produced
Evidence: Raw terminal transcript of the real grok run
Evidence: gnhf argv vs the real `grok --help` (grok 0.2.114)
/var/folders/92/rcfm4q4s51bcz30v1tw_vjl00000gn/T/no-mistakes-evidence/01KYSTS1VAGJKQVWGK8Z5HSVE6/gnhf-grok-live-tui.png) - Evidence: Exit summary after a 2-iteration fixture run (screenshot) (local file:/var/folders/92/rcfm4q4s51bcz30v1tw_vjl00000gn/T/no-mistakes-evidence/01KYSTS1VAGJKQVWGK8Z5HSVE6/gnhf-grok-exit-summary.png)Evidence: Fixture run: git log, README, notes.md, gnhf.log agent events
Evidence: --max-tokens stops after the iteration and keeps the commit
Evidence: Selecting grok via ~/.gnhf/config.yml with path and arg overrides
Evidence: grok discoverability in --help, error text, and bootstrapped config
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
e2e/e2e-grok.test.ts:155- The three new grok e2e tests run on every CI OS, but the grok spawn path cannot work on Windows.createTestEnvputse2e/fixtures(containing both an extensionlessgroksh script andgrok.cmd) on PATH, soshouldUseWindowsShell("grok", "win32")shells out towhere grok. If it resolves the extensionless file first,shellis false andspawn("grok")fails ENOENT (libuv only appends .com/.exe for extensionless names). If it resolvesgrok.cmd,shell: trueis used and Node'snormalizeSpawnArgumentsjoins[file, ...args]with plain spaces and no quoting - I confirmed this locally - so the multi-line-piteration prompt and the--json-schemaJSON blob are split into separate tokens, breakingexpect(invocation.prompt).toContain(...)andJSON.parse(invocation.schema). The existing opencode e2e passes on Windows only because all its argv tokens are single words and the prompt travels over HTTP. Sibling suites already gate this class of test withit.skipIf(process.platform === "win32")(e2e/e2e.test.ts:456, e2e/e2e-acp.test.ts:206). AGENTS.md/CONTRIBUTING require all three CI OS jobs green. Fix: gate the three grok e2e tests on win32 like the siblings.src/core/agents/grok.ts:241- Product-level counterpart of the e2e issue: on Windows, an npm-installedgrok.cmdshim makesshouldUseWindowsShellreturn true, and Node then builds the command line as[bin, ...args].join(" ")with no per-arg quoting. The concrete path is: Windows user installs the grok CLI as a.cmdshim ->buildGrokArgsemits-p <multi-line iteration prompt> --output-format streaming-json --json-schema {"type":"object",...}-> cmd.exe receives that unquoted, so grok gets-pwith only the prompt's first word and a shredded schema argument. The invariant that argv tokens survive the spawn boundary is violated. This is pre-existing and equally reachable for claude/codex/copilot/pi, and this change only moved the helper intostream-utils.tsrather than introducing it, so it is not a regression - but grok is a newly reachable instance of it. The earliest shared boundary for a durable fix is theshouldUseWindowsShell+ spawn call sites instream-utils.ts: resolve the shim path and spawncmd.exe /d /s /cwith explicit quoting pluswindowsVerbatimArguments, rather than relying on Node's unquoted shell join. Flagging for the author's decision since fixing it changes behavior for all five spawn agents.src/core/agents/grok.ts:186-parseGrokOutputgates onend.structuredOutput !== undefined, so anendevent that explicitly carries"structuredOutput": null(a common JSON wire convention for "no structured answer") is treated as present and sent straight tovalidateGrokOutput, which throwsInvalid grok output: .... That skips the streamed-text recovery path the function was written for, so a grok build that emits an explicit null instead of omitting the field would fail every iteration despite printing a valid JSON answer as prose. Widening the guard toend.structuredOutput != nullcosts nothing and keeps the documented fallback reachable.src/core/agents/grok.ts:181-parseGrokOutputplus itsmatchesSchemahelper is the third near-verbatim copy of the schema-preferring parse-and-validate routine, afterparseCopilotOutput(src/core/agents/copilot.ts:148) andparseOpenCodeOutput(src/core/agents/opencode.ts:216); rovodev has a fourth variant at src/core/agents/rovodev.ts:749. All four do:parseAgentJson(text, schema-accepts)-> validate, elseparseAgentJson(text)-> validate, else throw"<agent> output did not contain a parseable JSON object". AGENTS.md already namesjson-extract.tsas the home for this ("use it before adding ad-hoc parsing"). Non-functional dedup: lift aparseAndValidateAgentOutput(text, schema, agentName)helper there and have the four agents call it.🔧 Fix: harden windows agent argv escaping and share output parsing
3 issues (1 warning, 2 infos) still open:
src/core/agents/stream-utils.ts:97- The Windows escaping is correct for spaces, quotes and operators, but it does not make the original failure unreachable:\r/\ncannot be caret-escaped for cmd.exe (a caret before a newline is line-continuation, which deletes it), and cmd terminates the /c command at an embedded LF. Concrete path: Windows user has claude/codex/copilot/grok installed as an npm.cmdshim ->shouldUseWindowsShellreturns true ->spawnAgentProcessescapes and handscmd /d /s /ca command line containing the always-multi-line iteration prompt from src/templates/iteration-prompt.ts -> the agent receives only the prompt's first line as-p, and the remaining lines are parsed by cmd.exe as further commands (prompt and notes text is attacker-influenceable in a loop that runs unattended overnight). Rust's std made exactly this call for batch files, rejecting args containing a newline as unescapable.piis unaffected because it sends the prompt over stdin. The e2e suite documents the limitation (e2e/e2e-grok.test.ts:155 comment: "cannot carry the multi-line iteration prompt as a single argv token no matter how it is quoted"), but the product code neither rejects nor works around it, and README.md:315 still states "On Windows,.cmdand.batwrappers are supported". Earliest shared boundary isspawnAgentProcessitself: refuse the cmd.exe route when any arg contains\r/\n- either resolve the shim's real target and spawn it directly, or carry the prompt via stdin/a prompt file the way pi already does - or at minimum fail fast with a clear error instead of silently truncating. Flagging for the author since the options differ in user-visible behavior and this is a pre-existing limitation the change improved rather than introduced.src/core/agents/stream-utils.ts:73-escapeWindowsShellArgumentreproduces cross-spawn'sescapeArgument(same metachar regex/([()\][%!^"<>&|;, ?])/g, same backslash/quote doubling) but omits itsneedsDoubleEscapeMetaCharsstep, which cross-spawn applies precisely when the target matchesnode_modules/.bin/.cmd- the caseshouldUseWindowsShell's own doc comment names ("npm installs some agent CLIs as.cmd/.batshims"). Mechanism: the shim re-expands%*intonode "<cli>.js" <args>, so cmd parses the argument text a second time; cmd's quote tracking toggles on every"and does not understand the&feat(agents): add Jules cloud agent with REST API client #34;escapes we emit, so roughly half of a quote-dense argument's segments are outside quotes on that second pass, and any&,|,<,>,(,)landing there is interpreted rather than passed through. The built--json-schemapayload happens to be safe (buildAgentOutputSchema emits no prose descriptions, and,/[/]/{/}are not cmd operators), so this is latent rather than always-hit today, but it becomes the live failure as soon as the newline issue above is addressed. Adding the secondreplace(WINDOWS_SHELL_META_CHARS, "^$1")` pass for shim targets restores parity with the reference algorithm.src/core/agents/rovodev.ts:313- stream-utils.ts:15 now statesshouldUseWindowsShellis "kept module-private so every caller goes throughspawnAgentProcess, which is the only place that knows how to keep argv intact once a shell is in play", but rovodev.ts:73 still holds a verbatim private copy of that function and rovodev.ts:313 spawns withshell: shouldUseWindowsShell(this.bin, this.platform)and unescaped argv - including...(this.extraArgs ?? [])fromagentArgsOverride.rovodev. So a rovodev override value containing spaces or metachars against a.cmd-shimmed acli is still split apart, and the two copies of the resolution heuristic can now drift. Route rovodev's server spawn throughspawnAgentProcessand delete the local copy (rovodev.ts:103terminateRovoDevProcessis likewise a copy ofterminateChildProcess). Note opencode.ts:471 deliberately uses a different policy (shell: isWindowsunconditionally) that its Windows e2e depends on, so it should not be folded in without separate thought.✅ **Test** - passed
✅ No issues found.
npx vitest run src/core/agents/(238 tests: grok.test.ts, factory.test.ts, json-extract.test.ts, stream-utils.test.ts, plus claude/codex/copilot/pi/rovodev/opencode tests affected by the shared-helper refactor)npx vitest run e2e/e2e-grok.test.ts(wire format + commit, --max-tokens keeps the finished commit, text-only JSON recovery)npx vitest run src/core/config.test.ts src/core/agents/factory.test.ts(grok config/factory plumbing)pnpm run build(required before e2e, which drives dist/cli.mjs)Manual end-to-end with the REAL grok CLI:node dist/cli.mjs "Add a JSDoc comment to the addTodo function in todo.js..." --agent grok --max-iterations 1in a scratch git repo -> real grok turn, real file edit, real gnhf commitManual wire-format cross-check: captured argv gnhf builds vsgrok --help(grok 0.2.114) for-p,--output-format streaming-json,--json-schema,--always-approve, and the--permission-mode/--allow/--deny/--disallowed-tools/--toolsallowlistManual multi-iteration run against the e2e grok fixture:node dist/cli.mjs "add one bullet to the README" --agent grok --max-iterations 2-> 2 commits, notes.md history, gnhf.log agent lifecycle eventsManual config-file path:~/.gnhf/config.ymlwithagent: grok+agentPathOverride.grok+agentArgsOverride.grokand no--agentflag -> grok selected, overrides land before gnhf's own flagsManual budget path:--agent grok --max-iterations 5 --max-tokens 100-> stops after the iteration with the commit kept, grok invoked exactly once, noagent:run:abortedeventManual discoverability:gnhf --help,gnhf x --agent grokkerror text, and the bootstrapped~/.gnhf/config.ymloverride examplesTerminal-stream replay + headless Chrome screenshots of the live TUI and exit summary for both the real-grok run and the fixture run✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.