UI Automation Agent

An intelligent AI virtual assistant that automates computer interactions. Using Gemini as the brain and Computer Vision as the eyes, this agent can read and understand application interfaces, plan actions, and safely execute mouse and keyboard operations under human supervision.

Overview

UI Automation Agent acts as an autonomous, vision-language-driven agent that interacts with any computer interface exactly like a human would: by looking at the screen, reasoning about the layout, and taking actions.

Powered by Gemini 3.1 Flash, this project bridges the gap between Large Vision-Language Models (VLMs) and actual OS-level execution.

✨ Key Features

• Set-of-Mark (SoM) Visual Grounding: Instead of blindly guessing coordinates, the agent dynamically detects UI elements (via OCR and edge detection) and overlays unique IDs on the screen. This guarantees zero-hallucination clicks.
• The ReAct Framework Loop: The core engine follows a strict Observe -> Reason -> Act loop, allowing the agent to self-correct and recover from errors (e.g., a missing button or failed click).
• Human-in-the-Loop (HITL) Safety: AI can be dangerous. For irreversible actions (like sending emails, deleting files, or submitting payments), the agent utilizes native OS-level popups to explicitly ask for human permission, blocking the execution thread securely without polluting the OCR context.
• Smart UI Synchronization: Replaces hardcoded time.sleep() with intelligent, SSIM-based (Structural Similarity Index) computer vision checks. The agent naturally "waits" for loading spinners to finish and web pages to stabilize before making its next move.

Illustration of the Agent with the prompt: “Search for ‘OpenAI’ on Google and click the first link on the search results page.”

You can also check out some more example prompts and their corresponding agent behaviors in the public/demo_cases directory.

Prerequisites

Before you begin, ensure you have the following installed:

• Python 3.11 or higher.
• Google Gemini API Key: Get it from Google AI Studio.
• Operating System:
  • Windows 10/11: Ensure your Display Scaling is set to 100%.
  • macOS: Before running the agent, you must grant the following permissions to your Terminal or IDE:
    1. Screen Recording: Required for the agent's visual perception.
    2. Accessibility: Required for simulating mouse clicks and keyboard strokes.
  • Linux: Must use X11 display server (Wayland is currently NOT supported due to pyautogui and screen capture limitations).

Installation

1. Install uv (if you haven't already):

# On macOS and Linux:
curl -LsSf [https://astral.sh/uv/install.sh](https://astral.sh/uv/install.sh) | sh

# On Windows:
powershell -ExecutionPolicy ByPass -c "irm [https://astral.sh/uv/install.ps1](https://astral.sh/uv/install.ps1) | iex"

2. Clone the repository and navigate to the project directory:

git clone https://github.com/lilkhoa/UI-Automation-Agent.git
cd UI-Automation-Agent

3. Activate the virtual environment:

uv venv

# On Windows:
.venv\Scripts\activate
# On macOS and Linux:
source .venv/bin/activate

4. Install the required dependencies:

# Install core dependencies:
uv pip install -e .

# OR, install with development tools:
uv pip install -e ".[dev]"

5. Environment Setup: Create a .env file in the project root and add your Gemini API key:

GEMINI_API_KEY=your_gemini_api_key_here

Usage

To start the UI Automation Agent, first you must create a data directory in the project root with the following structure:

data/
├── screenshots/        # Stores screenshots taken by the agent
├── logs/               # Stores logs of agent actions and observations
├── models/             # Stores OCR models
└── videos/             # Stores screen recordings of agent sessions

1. Quick Start

To give the agent a task, simply edit the src/main.py file and modify the user_goal variable with your desired instruction. For example:

user_goal = "Search for 'OpenAI' on Google and click the first link on the search results page."

Then run the agent:

python src/main.py

2. Expected Output

Once running, you will see the agent's thought process and actions logged directly to your terminal. It will capture the screen, annotate it, query the VLM, and execute OS-level commands.

19:13:09 | INFO     | Step 1/15
19:13:10 | INFO     | Annotated 35 UI elements
19:13:10 | INFO     | Requesting action from VLM for goal: Search for 'OpenAI' on Google and click the first link on the search results page.
19:13:13 | INFO     | VLM Decision: type | Thought: The user wants to search for 'OpenAI' on Google. I need to click on the search bar (ID 19) to focus it, then type 'OpenAI' and press enter.
19:13:14 | INFO     | Clicked at (833, 566)
19:13:14 | INFO     | Focused on ID 19 before typing
19:13:15 | INFO     | Typed text: 'OpenAI'
19:13:15 | INFO     | Typed: OpenAI

3. Reviewing Logs & Video Recordings

By default, the agent is configured to record its entire workflow for traceability and debugging. After a run completes (or fails), check the data/ directory:

• data/logs/: Contains detailed .log files with full JSON responses and debug traces.
• data/videos/: Contains .mp4 video recordings showing the mouse movements and UI interactions.

4. Configuration

You can also customize the agent's behavior by modifying the parameters in config.yaml or directly when initializing the UIAgent class.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Made by lilkhoa