UAA - Unity AI Assistant

Utilize LLMs to develop in natural language inside Unity engine.

Youtube   •   LinkedIn

Description

UAA is a tool that integrates Large Language Models (LLMs) with the Unity engine, enabling developers to execute commands using natural language whether you're using open-source models locally with LM Studio or paid models through OpenAI.

Features

UAA Chat

• Chat directly with the AI Assistant within Unity, eliminating the need to switch between different platforms (back and forth to ChatGPT, Bing or any other platform out there...)
• The chat has everything you expect from a LLM Chat Bot like saving / loading chat history, automatic naming of the chat...
• Upcoming: Scene-awareness feature to provide context-specific assistance based on the current open scene and project.

UAA command

• Issue natural language commands to the AI Assistant, which converts them into executable Unity scripts.
• UAA mitigates LLM inconsistencies by checking the console for errors and making corrections automatically.
• Upcoming: Self-validation and user validation features to ensure scripts deliver the desired outcomes.

UAA Profile

• Customize the AI Assistant's behavior by adjusting prompts, iteration count, and request settings.

---

Try It

"YoutubeLink.com"

Installation

Unity Asset Store

Link to the asset store (coming soon)

Git URL

Project supports Unity Package Manager. To install the project as a Git package do the following:

1. In Unity, open Window -> Package Manager.
2. Press the + button, choose "Add package from git URL..."
3. Enter "GithubUrl.com" and press Add.

Manual

Copy the Assets/Editor/UAA folder to your project's Assets/Editor directory.

Setup

Local open-source models using LM Studio

• Download LM studio: LM Studio - Discover, download, and run local LLMs
• In LM Studio Download Mistral-7B-Instruct-v0.2-GGUF or any model that you would like to try: TheBloke/Mistral-7B-Instruct-v0.2-GGUF · Hugging Face
• Set up the model settings and the Server configuration and Start Server.

Paid models OpenAI

• Head to API keys - OpenAI API
• Create a new secret key - API Key
• in Assets/Editor/UAA/ProtectedEnv/UAAProtectedEnv.cs past your OPENAI_API_KEY.

---

Diagrams

UAA Chat

UAA Task

UAA Command

---

Roadmap

• Dev-Env Awareness: Develop a feature for UAA that understands the current development environment (Scene Object Hierarchy and the project as whole.)
• Project Embeddings: Explore embedding the entire project repository into the model for contextual understanding.
• Train / Fine-Tune a Model: Train the LLM on the Unity API and fine-tune it with command-to-task-to-script examples.
• Safe Execution Checkpoints: Establish a system for saving temporary scene states before script execution.
• User Task Validation: Implement a feature where the LLM provides a task description and script for user validation and feedback.
• Self-Validation: Create a self-validation mechanism instead of User Task Validation.
• Chat History Summarization: Add a function to summarize extensive long chat histories.
• Script Repository: Save successfully executed and validated scripts for future reference.
• UAA Debugger.
• Enhanced Assistant Responses: Improve quality by initializing the assistant response.
• Additional Detail Inquiry: Enable the LLM to request more task details when necessary.
• Prompt Guidelines: Establish guidelines for consistent responses.
• Voice Commands.
• Add amuck display that takes ins consideration things like bold code highlight.

---

Author

Mehdi Ali : AliMehdi.offi@gmail.com