feat(computer): full-desktop computer-use worker - #494
Closed
rohitg00 wants to merge 10 commits into
Closed
Conversation
computer::* drives a whole desktop over the bus through a computer-server backend: connect a session, screenshot it as image content blocks the vision model renders inline, act by coordinate (click/right_click/double_click/move/drag/scroll/type/press/hotkey), run shell, and read/write files in the guest. Sessions are durable via state and reconnect best-effort on restart; the live screen streams onto computer:frames (stream::set) for the console and any watchers; session-started/-stopped are custom trigger types. The harness auto-discovers the surface as tools. Backend is behind a Backend trait (v1: the computer-server WS wire; a host-background driver can slot in later). Wires release (create-tag + release workflows), the modules table, and the console-plumbing permission denies. Golden wire-schema tests cover all 12 functions; fmt + clippy -D warnings + tests green.
- config: drop the unwired default_timeout_ms/clamp_timeout; the sole action timeout is command_timeout_ms, fixed at connect (was a no-op knob that silently did nothing) - perf: store the screencast frame as Arc<LatestFrame>, removing two multi-MB base64 copies per produced frame and per no-change poll; clear the buffer on screencast stop - dedupe: Session::new constructor, a shared side_write helper (persist/forget now log at warn since a lost record breaks restart-restore), ContentBlock::image/text + Shot::to_base64/byte_len, a CommandError enum over the (bool,String) pair, command_ok for the action methods; shutdown now reuses stop_screencast - honesty: drop the unreachable 'crashed' reason from the event docs; remove the unused url dependency carried from the browser mirror
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
skill-check — worker0 verified, 43 skipped (no docs/).
Four for four. Nicely done. |
…ktops sessions::start with no endpoint now drives the local machine the worker runs on, via a native backend (xcap for screen capture, enigo for mouse/keyboard) behind the same Backend trait. An endpoint still routes to computer-server for a remote or sandboxed desktop. This answers 'why need computer-server when iii is running': for the local machine, iii drives it directly with zero external server; computer-server is only for desktops that live elsewhere. The second Backend impl also validates the trait the review flagged as single-impl. macOS needs Accessibility (input) and Screen Recording (capture) granted to the worker process. Golden regenerated for the sessions::start description change; fmt + clippy -D warnings + 25 tests green.
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Maximum reuse of existing workers. computer::shell and computer::files::read/write duplicated the shell worker (shell::exec, shell::fs::*, coder folded in), so they are removed. The computer worker now owns only what no other worker can do: capture the screen and drive the cursor. Surface goes from 12 to 9 functions (sessions, screenshot, observe, act, screencast, frame). Shell and files on the desktop compose with the shell worker; state via state::*; desktop/VM lifecycle stays with the sandbox worker (computer only connects to a computer-server or drives the native host, it never boots a VM). Removed the run_command/read_text/write_text backend methods and CommandOutput. fmt + clippy -D warnings + tests green.
A full Retina screenshot was tens of MB of PNG (~7MB base64 each), flooding the model context and the frame stream. Native captures are now downscaled to max_screenshot_dimension (default 1280 longest edge) and JPEG-encoded (default quality 70), cutting a screenshot from ~7MB to ~100-200KB. The model sees the downscaled image and its coordinate space; pointer actions scale coordinates back up to real screen pixels before injecting (the native backend learns the real display size on first capture). Two new config knobs (both native-only), two tests. fmt + clippy -D warnings + 26 tests green.
Adds a chat/computer/ renderer mirroring chat/browser and chat/scrapling: computer::screenshot and computer::observe render their image content blocks as an inline <img> + caption, and sessions/act get compact lines. Wired into FunctionCallCard (the single dispatch point, also feeding TracesV2), following the same ?? ChainedToolView pattern after the coder family. Screenshot parsing mirrors the browser worker's parseScreenshotOutput (reads content at the top level, accepts details flat or nested under the harness) rather than unwrapEnvelope, which would discard the image blocks. Console typecheck passes. Needs a console rebuild/reinstall to take effect in a running console.
xcap/enigo pull libdbus-sys + xcb, whose system libs headless Linux CI and servers lack, so the build failed there. The native (drive-this-machine) backend only makes sense on a desktop OS, so xcap/enigo/image and the native module are now target-gated to macOS/Windows; other targets build with the computer-server backend only, and sessions::start with no endpoint returns a clear 'native backend is only available on macOS and Windows' error. Verified via cargo tree: Linux graph has zero native deps, macOS keeps them. macOS fmt + clippy + 26 tests green.
Without macOS Accessibility, synthetic input is silently dropped and enigo still reports success, so act returned ok:true for a click that never landed. Native input now preflights AXIsProcessTrusted and returns a clear 'grant Accessibility to this process' error instead of a false success. Screen Recording is deliberately NOT preflighted: CGPreflightScreenCaptureAccess false-negatives for a worker running as a child of a granted terminal (it reported not-granted for a capture that actually works, which broke sessions::start), so a missing grant instead surfaces as a wallpaper-only image. macOS FFI, stubbed true on Windows. fmt + clippy -D warnings + tests green.
The native backend captured an arbitrary display (Monitor::all().next()) and mapped clicks in backing pixels, so on a Mac with an external monitor it grabbed the wrong screen, and on Retina it clicked at 2x the intended point. Now it captures the display under the cursor by default (follows where you work), pinned by id per session for stable coordinates, and maps downscaled coordinates through the display's LOGICAL point size and global origin (CGDisplayBounds, the same space enigo absolute coordinates use) so clicks land correctly at any scale factor and on any display. Adds computer::displays to enumerate displays live and an optional monitor index on sessions::start. Nothing is hardcoded: displays are enumerated at runtime, the cursor position is queried, and each display's geometry (origin, logical size, scale) is read from the OS. 10 functions; goldens regenerated; fmt + clippy -D warnings + tests green.
…socket) Third Backend, IiiSandboxHost: boots a desktop inside an iii-sandbox microVM and drives it entirely through iii primitives (sandbox::exec + sandbox::fs). iii-sandbox has no host-to-guest TCP, so there is no computer-server and no inbound socket; the display is driven by exec. - sessions::start gains `image`: boot a fresh desktop from an OCI image (a sandbox preset or custom_images key). Precedence: image > endpoint > native. - bootstrap starts Xvfb + openbox detached with setsid (survives between exec calls and escapes the daemon's process-group SIGKILL on disconnect), waits for the display, then screenshots via `import | base64` and injects input via xdotool under DISPLAY=:0. Pointer/keyboard run as a raw argv, so caller text is never shell-interpreted. - fixed virtual resolution (config sandbox_width/height, default 1280x800): 1:1 coordinates, no HiDPI or multi-monitor ambiguity, no host permissions. - durable: the session persists as `sandbox:<id>`; reconnect re-attaches and re-runs the idempotent display bootstrap, dropping the record if the VM is gone. - config: sandbox_image, sandbox_width, sandbox_height, sandbox_idle_timeout_secs (high, so the sandbox reaper never kills a live desktop; the worker owns teardown). - images/desktop: Dockerfile + README for the desktop image. Verified against iii-hq/iii crates/iii-worker/src/sandbox_daemon: macOS Apple Silicon boots (libkrun/Hypervisor.framework), images are OCI refs, the image ENTRYPOINT is not run (so the worker starts the display), and setsid-detached processes survive across exec calls.
Contributor
Author
|
Superseded by #673, which is merged. That branch was 342 commits behind main and predates the native and sandbox drivers; the worker landed with three drivers, its own injected console page, and the release wiring. Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear: MOT-4001
What
A new
computerworker: full-desktop computer use over the iii bus, the sibling ofbrowser. Wherebrowsergives an agent a Chromium tab,computergives it a whole screen. The harness auto-discoverscomputer::*as tools, hands the model a screenshot, and the model clicks, types, scrolls, and runs shell inside the desktop by coordinate.The desktop itself lives outside iii (booting real macOS/Windows/Linux desktops is deep systems work). The worker connects to a
computer-serverendpoint, the in-guest executor that open desktop-sandbox stacks expose (a Cua sandbox is one such backend), and speaks its{command, params}WebSocket wire. iii owns the session lifecycle, the bus, durable state, the live screen stream, and the harness that drives it.Surface (12 functions, golden wire-schema tested)
computer::sessions::start/list/stopcomputer::screenshot(image content blocks the vision model renders inline),computer::observe(screenshot + optional accessibility tree)computer::act(click, right_click, double_click, move, drag, scroll, type, press, hotkey, by pixel coordinate)computer::shell,computer::files::read/writecomputer::screencast::start/stop+computer::frame(internal console-UI plumbing, denied to agents)computer::session-started/computer::session-stoppedWhat iii adds over a one-shot computer-use client
state; on boot the worker reconnects them best-effort, so a restart does not lose live desktops.computer:framesstream (stream::set), for the console and any number of watchers.Design notes
Backendtrait; v1 ships one implementation (ComputerServerClient, the computer-server WS wire). A host-background driver is a different integration shape and can slot in behind the same trait later.Testing status
cargo fmt --check,cargo clippy --all-targets --all-features -- -D warnings, andcargo testall pass (25 tests: golden wire-schema for all 12 functions, plus config, manifest, backend, and events).Marked draft: not yet validated end to end against a live computer-server and desktop sandbox. That is the next step before it is ready to merge.
Follow-ups (separate PRs)
Computerpage: a fork of theBrowserpage reusing the screencast viewport and coordinate input.sessions::startcan boot a desktop, not only attach to a running one.