An SDK for agentic UI testing in Xcode. Describe what a test should accomplish in plain English — UIXCute drives your app using XCUIAutomation, powered by an AI agent running on AWS Bedrock or a local Ollama model.
func testCheckout() async throws {
let app = XCUIApplication()
app.launch()
let report = try await UIXCuteAgent(app: app).run(goal: """
Add the first item to the cart, proceed to checkout,
fill in the shipping address, and verify the order summary is shown.
""")
// report.steps lists every tool the agent called, in order
}Each call to run(goal:) or step(_:) enters an agentic loop:
- UIXCute captures the current UI state (accessibility tree and/or screenshot).
- It sends the state, the goal, and a set of available tools to the LLM backend (Bedrock or Ollama).
Screenshots are delivered as native vision content blocks — the model sees the actual PNG image, not a text description. - The model responds with tool calls —
tap,typeText,scroll, etc. - UIXCute executes those calls via XCUIAutomation and feeds the results back.
- The loop continues until the model calls
terminate(success:)or a limit is hit.
When screenshots are enabled, only the most recent screenshot is included in each request. Older screenshots are replaced with a [screenshot replaced by latest] placeholder so the full action history is preserved without quadratic token growth.
iOS system permission alerts (location, camera, photos, notifications) are handled automatically. These prompts are drawn by SpringBoard, a separate process, so their buttons never appear in the app's own accessibility tree. UIXCute detects an active system alert during context capture, surfaces its buttons to the model as a ## System Alert (SpringBoard) section, and routes a tap to the alert when the app has no matching element. The agent can grant or dismiss the prompt just by tapping a button label — no screen coordinates and no vision model required:
try await agent.step("Tap the Allow Location Access button")
try await agent.step(#"Tap "Allow Once" to grant location access"#)- Xcode 26+ with Swift 6.2
- An AWS account with access to Amazon Bedrock and the Anthropic Claude model family enabled in your chosen region (Bedrock backend), or a locally running Ollama instance (Ollama backend)
- Add UIXCute as a dependency to your UI test target (not the app target)
In Package.swift:
dependencies: [
.package(url: "https://github.com/Andrea-Scuderi/UIXCute", from: "1.0.0-alpha.1"),
],
targets: [
.testTarget(
name: "MyAppUITests",
dependencies: ["UIXCute"]
),
]Or in Xcode: File › Add Package Dependencies, search for the repository URL, then add UIXCute to your UI test target.
UIXCute supports two LLM backends:
- AWS Bedrock setup — default backend; credentials, model selection, XCTestPlan configuration, and CI secrets management
- Ollama setup — run with a local LLM (llama3.2, llava, qwen3, gemma3, …); no AWS account required
Provide a single goal. The agent plans and executes all steps on its own.
import XCTest
import UIXCute
class LoginUITests: XCTestCase {
func testLogin() async throws {
let app = XCUIApplication()
app.launch()
try await UIXCuteAgent(app: app).run(goal: """
Log in with email user@example.com and password secret123.
Verify the home screen is shown after login.
""")
}
}Drive the agent one instruction at a time. Use assert(_:) for explicit checkpoints.
func testLoginStepByStep() async throws {
let app = XCUIApplication()
app.launch()
let agent = UIXCuteAgent(app: app)
try await agent.step("Tap the Email text field and type user@example.com")
try await agent.step("Tap the Password field and type secret123")
try await agent.step("Tap the Log In button")
try await agent.assert("The home screen is visible with a welcome message")
}Step-by-step agents share conversation history across calls, so each step knows what happened before.
Use step(_:expecting:) to supply an explicit success criterion — useful when a button navigates away (so the element disappears post-tap) or when a small model misreads a valid post-action screen as failure:
// Without expecting: model sees "Sign Out" gone and may report failure
try await agent.step("Tap Sign Out")
// With expecting: model judges success against the stated criterion instead
try await agent.step("Tap Sign Out", expecting: "The login screen is visible")Every run, step, and assert call returns an AgentExecutionReport — a structured trace of every tool the agent invoked, in order. The return value is @discardableResult, so existing tests compile unchanged.
let report = try await agent.run(goal: "Log in and verify the home screen")
print("Outcome: \(report.outcome)") // .success(reason:) or .failure(reason:)
print("Duration: \(report.duration)s")
for step in report.steps {
print("[\(step.stepIndex)] \(step.toolName)")
print(" input: \(step.input)")
print(" result: \(step.resultSummary)")
}Example output:
[0] tap
input: ["label": "Email"]
result: Tapped element (label=Email)
[1] typeText
input: ["identifier": "email_field", "text": "user@example.com"]
result: Typed 'user@example.com' into field (identifier=email_field)
[2] terminate
input: ["success": true, "reason": "Home screen is visible"]
result: Test terminated successfully.
On failure the report is always available via lastExecutionReport, so you can inspect the partial trace even when the agent throws:
do {
let report = try await agent.run(goal: "...")
attachReport(report)
} catch {
if let report = await agent.lastExecutionReport {
attachReport(report) // partial trace up to the failure
}
throw error
}Attaching to Xcode test results — use XCTAttachment to surface reports in the Xcode test results viewer so QEs can audit every run without reading raw logs:
private func attachReport(_ report: AgentExecutionReport) {
let text = formatReport(report)
print(text) // appears in the Xcode console
let attachment = XCTAttachment(string: text)
attachment.name = reportTitle(report) // "✓ Log in and verify…" or "✗ …"
attachment.lifetime = .keepAlways
add(attachment)
}See DemoApp/DemoAppUITests/DemoAppUITests.swift for a complete implementation of attachReport, formatReport, and reportTitle helpers, and wrapper methods that capture reports automatically for every run, step, and assert call.
let config = UIXCuteConfiguration(
maxSteps: 60, // max agent loop iterations
stepTimeout: 10, // seconds per XCUI action
totalTimeout: 300, // total wall-clock budget
perceptionMode: .screenshotAndAccessibilityTree // highest fidelity
)
// Bedrock with an explicit model and region
let agent = UIXCuteAgent(
app: app,
configuration: config,
backend: .bedrock(
config: BedrockConfig(
modelId: "us.anthropic.claude-sonnet-4-6-v1:0",
region: "us-est-1"
)
)
)The Bedrock model and region are resolved inside BedrockConfig.init: explicit argument → environment variable (BEDROCK_MODEL_ID / AWS_REGION) → built-in default. See Bedrock setup for recommended model IDs. For local inference, pass a .ollama backend — see Ollama setup.
| Mode | What the model sees | Token cost |
|---|---|---|
.accessibilityTree (default) |
Element hierarchy as text | Low |
.screenshotOnly |
PNG screenshot (latest only per request) | Medium |
.screenshotAndAccessibilityTree |
Both (latest screenshot only per request) | High |
.screenshotFallback |
Accessibility tree when available; screenshot only when the tree is empty | Low → Medium |
.screenshotFallback is the recommended mode for apps with a mix of standard and custom-rendered views — most screens are handled cheaply via the accessibility tree, and only screens where the tree is empty (canvas views, game scenes, WebViews) incur a screenshot. Only the most recent screenshot is sent on each request; earlier ones are replaced with a text placeholder to keep costs linear with step count.
All failures throw UIXCuteError:
do {
try await agent.run(goal: "Complete the onboarding flow")
} catch UIXCuteError.assertionFailed(let reason) {
// A condition checked by assert(_:) was not met
} catch UIXCuteError.agentTerminatedWithFailure(let reason) {
// The agent decided it could not complete the goal
} catch UIXCuteError.timeout(let seconds) {
// Exceeded totalTimeout
} catch UIXCuteError.maxStepsExceeded(let steps) {
// Exceeded maxSteps — try increasing it or simplifying the goal
}- Be specific about the end state, not just the steps: "verify the confirmation banner is visible" is better than "tap submit".
- One scenario per
run(goal:)call — the agent has full context of all its prior actions within that call. - Use step-by-step mode when you need guaranteed checkpoints between actions.
- Start with
.accessibilityTree(the default). Use.screenshotFallbackfor apps with mixed standard and custom views — it uses the tree when available and falls back to a screenshot only when the tree is empty. Use.screenshotAndAccessibilityTreeonly when you always need both. - If the agent keeps getting lost, increase
maxStepsor break the goal into multiplestep(_:)calls. - iOS permission alerts are handled for you — just instruct the agent to tap the button you want (e.g. "Tap Allow Once"). No need to mention coordinates or that the alert is system-drawn.
UIXCuteAgent ← public entry point (#if canImport(XCTest))
└── AgentLoop ← agentic loop, XCTest-free, protocol-driven
├── LLMClientProtocol
│ ├── BedrockAgentClient ← SotoBedrockRuntime Converse API
│ └── OllamaAgentClient ← URLSession POST /api/chat
├── UITool implementations (tap, typeText, scroll, waitForElement, terminate)
└── ActionExecutorProtocol
└── XCUIActionExecutor ← live XCUI calls (MainActor)
AgentLoop and all tool implementations depend only on ActionExecutorProtocol and LLMClientProtocol, making them unit-testable without a simulator via MockActionExecutor. See agent-loop.md for a full Mermaid diagram of the loop internals.
Apache License 2.0