OpenGameAgent

Build AI NPCs and in-game agents that understand context, run ReAct tool loops, plan complex tasks, and execute reliable actions.

Website · Getting started · 简体中文

OpenGameAgent is an open-source C# agent runtime for AI NPCs and other agents that operate inside games. It is not an AI coding agent or a game generator. It equips in-game characters and systems with structured-context understanding, ReAct reasoning and tool use, complex-task decomposition, durable planning that adapts to new evidence, and reliable execution through game-owned tools. The game remains authoritative over every state change.

What is OpenGameAgent?

OpenGameAgent turns a model into a programmable NPC or in-game agent instead of a dialogue endpoint. Developers choose a model, provide structured context, register game-owned tools, compose extensions, stream progress, tool calls, and answers as they are produced, and steer or cancel active work. The compact kernel handles the bounded model/tool loop; the optional game runtime adds sessions, actors, game timelines, memory, plans, multi-NPC scheduling, engine-thread handoff, and durable action receipts.

Short tasks can run as a ReAct cycle: observe the current state, decide what to do, call one or more tools, inspect their structured results, and either act again or answer. Complex tasks can be decomposed into persistent goals and ordered plans that survive later inputs, wait for game events, preserve completed steps, and replace unfinished work when new evidence invalidates the original approach. Deterministic workflows remain available when the execution graph should be fixed rather than model-directed.

OpenGameAgent runs inside Godot or Unity, behind a native Unreal Engine client, or in a .NET game server or sidecar. Inputs are bounded JSON plus optional durable image observations, so they can represent dialogue, combat state, simulation ticks, UI events, sensor data, screenshots, or any other game-owned context; natural language is optional. No model is bundled, and both cloud and local API endpoints are supported.

From dialogue to reasoning, planning, and action

A dialogue-only character receives a prompt and returns a line. An AI NPC or in-game agent receives a goal and the current environment, reasons about the next step, uses developer-defined tools, observes authoritative results, and adapts its approach. Those tools are ordinary game operations: move, inspect, trade, build, schedule, recruit, investigate, or any other capability the developer exposes.

Dialogue-only character	Agent-driven character
Reads the latest dialogue	Observes bounded JSON containing dialogue, world state, events, UI input, sensor data, or simulation ticks
Produces the next line of text	Streams text and typed tool calls, then consumes structured tool results
Stops after one model response	Runs a bounded ReAct loop across model responses, tool calls, and structured results
Has no durable task model	Can decompose a complex goal into persistent steps and replan unfinished work as conditions change
Treats generated text as the outcome	Requests actions while game code validates permissions, rules, revisions, and state changes
Usually follows wall-clock chat history	Can reason against game time, timelines, save/session identity, actor identity, and scoped memory
Models one conversation at a time	Serializes each actor while allowing bounded concurrency across many NPCs
May repeat a write after a timeout	Can journal state-changing intents and reconcile authoritative receipts before retrying

OpenGameAgent is model- and provider-neutral. Characters operate through developer-defined tools; a model never directly controls game state. The game remains authoritative at every mutation boundary.

Not every interaction needs the full loop. A game can route greetings and factual replies through the side-effect-free Quick path, use the Agent route for short tool-driven work, promote complex tasks into persistent plans, and select deterministic workflows for fixed execution graphs.

Current source pre-release: 0.3.0-alpha.4. Public APIs can change before 1.0; pin an immutable tag or source commit for shipped games.

The kernel boundary is intentionally small and designed to stabilize early. New game-specific capabilities should normally arrive as extensions, tools, policies, workflows, or game-owned services instead of expanding the model/tool loop.

Install

Published versioned artifacts remain available on the Releases page. For the current 0.3.0-alpha.4 source line, C# and Godot development should use a pinned source checkout and reference only the projects the game needs.

The release pipeline binds every artifact to one source commit and generates RELEASE_MANIFEST.json plus SHA256SUMS.txt for verification. Runtime Protocol v1 uses capability negotiation for additive features; changes to required fields, enum meaning, cursor semantics, or lifecycle ordering require a new protocol version.

Unity 6 projects can install the complete 0.3.0-alpha.4 package directly from its immutable GitHub UPM tag:

https://github.com/EricSun0218/OpenGameAgent.git#upm/0.3.0-alpha.4

The same release is available through OpenUPM: openupm add com.opengameagent.runtime@0.3.0-alpha.4. The generated upm branch contains the tested binaries; the Unity source directory on main is not itself a distributable package.

<ItemGroup>
  <ProjectReference Include="path/to/OpenGameAgent/src/OpenGameAgent/OpenGameAgent.csproj" />
  <ProjectReference Include="path/to/OpenGameAgent/src/OpenGameAgent.Memory/OpenGameAgent.Memory.csproj" />
  <!-- Optional: in-process BGE-M3 INT8 embeddings; model weights stay game-owned. -->
  <ProjectReference Include="path/to/OpenGameAgent/src/OpenGameAgent.Memory.Onnx/OpenGameAgent.Memory.Onnx.csproj" />
  <ProjectReference Include="path/to/OpenGameAgent/src/OpenGameAgent.Attachments.Local/OpenGameAgent.Attachments.Local.csproj" />
  <ProjectReference Include="path/to/OpenGameAgent/src/OpenGameAgent.Media/OpenGameAgent.Media.csproj" />
  <ProjectReference Include="path/to/OpenGameAgent/src/OpenGameAgent.Persistence/OpenGameAgent.Persistence.csproj" />
</ItemGroup>

Adjust the paths relative to your game project and omit optional projects you do not use. The kernel, persistence, providers, memory, attachments, plugins, and engine-compatible client are also shipped as separate OpenGameAgent.* release artifacts. See Getting started and Engine integration before connecting a game.

Why do AI NPC agents need a game-specific runtime?

General agent loops commonly assume one user, wall-clock time, and a linear conversation. A game may have multiple timelines, save forks, thousands of actors, offline time jumps, engine main-thread constraints, and actions that must never be repeated after an uncertain failure.

OpenGameAgent keeps the reusable agent machinery independent from the game while exposing the coordinates games need:

• named timelines and integer ticks, with optional calendar JSON;
• structured observations and context slices with floating-point values intact;
• content-addressed screenshot/image input with decode validation, model-capability preflight, and session-authorized retrieval;
• preflight auto, side-effect-free quick, short-task direct/agent, persistent plan, and deterministic-workflow routes;
• host-derived execution scopes that keep auto/Quick/short Agent available while withholding persistent planning from unauthorized actors;
• per-actor serialization with bounded cross-actor concurrency;
• journaled action intents and authoritative game receipts;
• game-time-filtered, expiring, rankable memory with session/owner-partitioned durable storage;
• optional local/remote embeddings, partitioned rebuildable vector indexes, and single-authoritative-snapshot lexical/vector hybrid recall;
• skills selected by input type and available tools;
• input-aware tool visibility resolved before every model request;
• host-attested tool modes and durable, one-time, world-version-bound approval for high-risk calls;
• recurring game-time triggers and persistent actor mailboxes with payload-free backlog queries;
• a typed extension API for tools, skills, routes, workflows, hooks, events, and services;
• capability-aware model catalogs and developer-hosted short-lived credentials;
• opt-in local discovery and health profiles for Ollama, LM Studio, LocalAI, llama.cpp, and vLLM;
• lazy external-tool discovery and large-result artifact spill;
• Agent Plugins 1.0.0 packages containing portable skills and MCP servers;
• image, audio, and video generation through replaceable APIs;
• crash-aware generated-asset materialization and authoritative engine import;
• append-only traces, provider/framework/host timing attribution, benchmark reports, observation-only playback, and offline CI evaluation;
• realtime speech with barge-in, background-agent handoff, and replaceable presentation behaviors.

The runtime does not decide combat legality, inventory rules, economy changes, NPC permissions, or other business rules. The game exposes narrow tools, validates every requested mutation, performs it on the correct thread or server, and returns the authoritative receipt.

Architecture

Godot / Unity / Unreal Engine sidecar / .NET game server
        |
        | GameInput (bounded JSON + GameMoment)
        v
GameAgentRuntime
  context | skills | route | session | actor lane | extensions
        |
        v
compact stateful Agent kernel <---- steering / follow-up
  model stream -> tool calls -> tool results -> next turn
        |                              |
        |                              v
        |                    durable action dispatcher
        |                              |
        v                              v
model API                    game-owned validation + state

The kernel can also be used directly when a developer needs only a compact agent loop. The higher game runtime is a composition layer, not a required world schema.

Read Architecture for the ownership and failure boundaries.

What is implemented

Area	Capability
Agent kernel	Bounded ReAct model/tool loop, streaming typed messages, typed partial tool results, steering, follow-up, hooks, cancellation, strict transcript validation, provider failures as results
Tool execution	Provider-request schema preflight plus execution-time validation over a bounded JSON Schema subset, guaranteed result for every accepted call, ordered parallel epochs around sequential barriers, conflict-key serialization, exact-repeat loop protection, policy blocking/termination, host-attested explicit/task scopes, durable one-time approval, timeouts, uncertain write outcomes
Game runtime	Arbitrary JSON input, game clocks/timelines, auto/quick/direct/plan/workflow routing, shared per-input usage budget, optimistic sessions, duplicate-input protection, actor concurrency, active-run steering/abort
Realtime conversation	Bounded PCM16 streaming, live transcription/audio events, subtitle timing, barge-in cancellation/truncation, non-blocking background-agent handoff/steering, and cancel-replace presentation behaviors
Image input	PNG/JPEG/WebP/GIF admission, immutable content-addressed storage, reference-only transcripts, capability preflight, tool-result images, and authorized server retrieval
Extension API	Immutable builder; prompt/context/tool/skill/route/workflow/hook/provider/service registration; per-input tool visibility; typed lifecycle events and channels; namespaced persistent state
Official extensions	Tool policy, high-risk execution approval and search, structured player questions/recommended replies, goals, host-verified ordered task plans with dynamic replanning and durable pause/resume, memory, artifacts, knowledge, restart-resumable delegated agents with lineage/leases, tracing, and durable parallel workflow graphs
DevTools	Bounded JSONL recordings, named context-provider and staged memory-recall timing, provider/framework/host attribution, failure and durable-write metrics, concurrent benchmark runtime, local observation-only HTML playback, and offline/CI evaluation rules
World primitives	Durable actions, bounded engine-thread action handoff, resumable workflows, memories, skills, signals, game-time schedules, actor mailboxes with batch read-only pending status
Models and auth	Bundled capability/context/reasoning/cost directory, dynamic refresh, API-key/environment/stored/OAuth/local auth, developer-hosted short-lived credential gateway
External tools	Lazy on-demand search/describe/call by default; explicit direct exposure for small trusted catalogs
Portable plugins	Agent Plugins 1.0.0 plugin.json, immediate-child SKILL.md discovery, MCP stdio/Streamable HTTP, client namespaces, containment, and component-level failure isolation
Providers	Native Anthropic, Amazon Bedrock, Google Gemini/Vertex, Mistral, OpenAI Responses/Azure, OpenAI-compatible, OpenAI Realtime, Volcengine realtime speech, remote gateway, and message-gateway transports; retry/fallback decorators; provider-neutral conformance runner and fixtures; optional local discovery for Ollama, LM Studio, LocalAI, llama.cpp, and vLLM
Generated media	Provider-neutral image/audio/video registry, generic async HTTP jobs, OpenRouter previews, official OpenAI Images, Volcengine Ark/Seedream, plus optional LocalAI and trusted ComfyUI workflow adapters
Generated assets	Stable operations, content-addressed resources, persistent lifecycle state, explicit uncertain outcomes, resumable import, and durable authoritative engine receipts
Persistence	Crash-tolerant local session snapshots, cross-process coordination, action journals, ordinary-tool replay journals, generated-asset jobs/resources, workflow checkpoints, session/owner-partitioned memory with flat-layout migration, mailboxes, artifacts, delegations, skills, and prompt templates
Semantic memory	Optional model-agnostic embeddings, single-snapshot authoritative verification, partitioned rebuildable local vector indexes, lexical/vector hybrid recall, content-free stage metrics, and game-time reranking
Placement	Shared netstandard2.1 runtime in Godot, Unity, or another C# host; optional .NET 8 HTTP/SSE service with C# and native C++ clients
Runtime protocol	Optional versioned Session/Run/Turn/Item contract, capability negotiation, stable event IDs, bounded replay/gap reconciliation, exact run/turn control, C# client, Schema/fixtures, C++ DTOs, and generated TypeScript/Python clients and reducers
Engines	Godot 4.7 .NET and Unity 6 in-process packages; Unreal Engine 5.8 native C++ sidecar plugin

Realtime speech is an optional layer rather than a second game-authority path. The realtime transport can converse or transcribe, and can request reversible gaze, gesture, expression, or movement presentation. Planning and durable world mutations are handed to the same GameAgentRuntime and game-owned tools used by non-voice inputs. Optional OpenAI and Volcengine adapters share this contract; the Volcengine adapter keeps dialogue/VAD and streaming TTS separate from the authoritative agent loop. See Realtime conversations.

For an entirely local stack, the optional OpenGameAgent.Providers.Local package provides bounded endpoint discovery and health checks, OpenAI-compatible embeddings, composable VAD/STT/streaming-TTS speech, LocalAI image/video/TTS generation, trusted ComfyUI workflows, and explicit host-authorized model inventory/warmup/load/unload/acquisition. No model is bundled or downloaded implicitly, and unknown capabilities are not guessed. See Local models, speech, and media.

Run inputs, model content, tool catalogs, loops, queues, progress, and concurrency are bounded by explicit limits. Context admission runs before every model request, model and tool calls have deadlines, and large tool results can be retained as artifacts instead of repeatedly filling the prompt. Game-owned stores and rankers can replace the included in-memory or local-file implementations.

Model access without hand-wiring every provider

OpenGameAgent.Models.BuiltIn turns the bundled model directory into an executable runtime. It currently dispatches nine wire APIs across 27 provider definitions and hundreds of text/tool-capable models, applying provider-specific request formats, reasoning settings, compatibility flags, cost metadata, authentication, cancellation, and bounded response handling. Provider usage is priced from the resolved directory when the provider does not report cost, while unavailable pricing remains explicitly unknown rather than appearing free. The lower provider packages remain independently usable when a game wants an explicit model and endpoint instead of a directory.

For known hosted providers, use the directory-backed runtime for both route classification and the main agent. The low-level OpenAI-compatible adapter intentionally requires explicit protocol settings and does not guess a provider family from a URL or model name.

OpenGameAgent.Models.Auth.BuiltIn adds opt-in browser or device authorization flows for supported subscription providers. Public client registrations are never embedded in the framework: flows that require a client ID remain disabled until the game developer supplies one. Windows desktop hosts can add OpenGameAgent.Models.Credentials.Windows for bounded, atomic CurrentUser DPAPI persistence behind the same IGameCredentialStore; other platforms can provide their native secure-store implementation without changing authentication code. OpenGameAgent.ProviderTransport exposes only allowlisted, bounded response metadata to observers and never passes credentials or arbitrary response headers to tracing code. See Windows credential persistence.

Image, audio, and video generation use a separate model registry because generation jobs, previews, polling, and outputs are not chat completions. The framework ships the neutral registry, generic and provider-specific adapters, and a generated-asset pipeline that materializes validated outputs before asking the authoritative game to import them. Games can register local generators, additional APIs, content-policy gates, and engine importers without changing the agent kernel. See Generated assets.

Minimal kernel

using OpenGameAgent.Kernel;
using OpenGameAgent.Providers.OpenAICompatible;

var http = new HttpClient();
var provider = new OpenAICompatibleProvider(new(http, endpoint)
{
    ApiKey = Environment.GetEnvironmentVariable("MODEL_API_KEY")
});

var agent = new Agent(new AgentOptions(provider, "your-model")
{
    SystemPrompt = "You are an NPC. Use tools when the world must change."
});

using var events = agent.Subscribe((e, _) =>
{
    if (e.ModelEvent?.Delta is { } delta) Console.Write(delta);
    return default;
});

var result = await agent.RunAsync("What can you see?");

Game runtime

var runtime = new GameAgentRuntime(new GameAgentRuntimeOptions(provider, "your-model")
{
    Instructions = "Act only from supplied game state. Use tools for mutations.",
    ContextProvider = myGameContext,
    ToolProvider = myGameTools,
    SessionStore = mySessionStore
});

var input = new GameInput(
    sessionId: "save-42",
    actorId: "npc-blacksmith",
    type: "player_interaction",
    payloadJson: """{"intent":"repair","item":"sword","durability":0.35}""",
    moment: new GameMoment("main-world", tick: 18840),
    inputId: "interaction-9001");

var run = await runtime.RunAsync(input);

See the buildable living-world example, the offline generated-asset example, and Getting started.

Choose where it runs

• Inside a C# engine host: simplest single-player deployment for Godot .NET or Unity, with direct access to game context and no extra agent server. Suitable for BYOK or local endpoints. A provider key shipped in a client can be extracted.
• In the game server: best when the game already has an authoritative server. Run the same C# runtime beside game rules and persistence.
• Separate agent service: useful for Unreal, centrally paid inference, secrets, scaling, or independent updates. C# and native engine adapters call OpenGameAgent.Server over JSON/SSE and can steer or abort an active actor through authenticated control endpoints.

For developer-funded client inference, use a developer-controlled gateway that issues short-lived scoped credentials. The permanent upstream provider key stays on developer infrastructure; the framework supplies the client credential flow, while the game owns login, quotas, revocation, and abuse controls.

Placement does not change ownership: only game code decides whether an action commits.

Build and verify

Requirements: .NET SDK 8.0. Windows and Linux are supported for the shared runtime and server. Engine adapters currently target Windows editor verification.

dotnet restore OpenGameAgent.sln
dotnet build OpenGameAgent.sln -c Release --no-restore
dotnet test OpenGameAgent.sln -c Release --no-build --no-restore
./engines/godot/test-package.ps1 -GodotSharpDir <GodotSharp/Api/Debug>
./engines/godot/test-engine.ps1 -Godot <godot_console.exe> -GodotSharpDir <GodotSharp/Api/Debug>
./engines/unity/test-package.ps1 -UnityManagedDir <Unity/Editor/Data/Managed/UnityEngine>
./engines/unity/test-editor.ps1 -UnityEditor <Unity.exe> -UnityManagedDir <Unity/Editor/Data/Managed/UnityEngine>
./engines/unreal/test-package.ps1
./engines/unreal/test-plugin.ps1 -UnrealRoot <UE_5.8>

Real-editor gates are documented in Engine integration.

Documentation

• Getting started
• Architecture and authority boundaries
• Feature and API map
• Game integration patterns
• Extension development kit
• Provider conformance
• Runtime Protocol and cross-language SDKs
• Engine integration
• Deployment and security
• High-risk tool approval
• Tool execution safety and concurrency
• Generated media
• Local models, speech, and media
• Generated assets and authoritative import
• Image input and game perception
• Execution routing and performance
• Traces, playback, and offline evaluation

What does OpenGameAgent provide?

This repository is a runtime framework for agents inside a game, not an AI coding agent that develops the game for you. It does not define a universal character sheet, combat model, world-package format, visual editor, or end-user game. Those belong to each game. OpenGameAgent provides the reasoning loop and game-aware primitives needed to build conversational NPCs, autonomous companions, social simulations, AI directors, generated quests and items, strategy agents, construction agents, and interactive worlds.

Attribution

Every distributed game, mod, application, or product that includes OpenGameAgent must make the OpenGameAgent copyright and MIT license notice available in Credits, About, Third-party Licenses, documentation, or an accompanying license file. The following concise credit may be used alongside the license notice: “Powered by OpenGameAgent | opengameagent.com”. See the license and brand assets and attribution guidance.

License

MIT License. You may build proprietary games and hosted products with the framework. Distributed copies must include the copyright and permission notice. See CONTRIBUTING.md and SECURITY.md.