Repo Overview

This repo the following reusable library projects.

• RTOpenAI.Api - A Maui library to interface with the OpenAI realtime voice API via the WebRTC protocol.
  • RTOpenAI.Events - strongly typed wrappers for OpenAI realtime API events
  • Bindings/FsWebRTC.Bindings.Maui.Android - Maui WebRTC bindings for Android
  • Bindings/FsWebRTC.Bindings.Maui.iOS - Maui WebRTC bindings for iOS
• RTFlow - A framework for building realtime multi-agents systems.
• FsPlan - A library for organizing a collection of tasks into a plan with either linear or graphical flow
• FsPlay - A Maui library for programmatically driving embedded mobile browsers, e.g. using computer-use-agents or CUA
  • FsPlay.Abstractions - Abstractions for browser automation
• FsAICore - Common AI utilities used by the other libraries here.

Samples Included

The libraries are complementary i.e. they can be combined together to build various types of AI applications. The use of these libraries are showcased in the following sample projects:

RT.Assistant

Please read the RT.Assistant sample write-up included in this repo. It showcases the use of these libraries in a realtime voice-assistant app.

Two videos are also available: Overview; Code Walkthrough.

RTOpenAI.Sample

A minimal sample showcasing RTOpenAI for interfacing with the OpenAI realtime voice API

RTOpenAI.Api Quick Usage

The normal case still uses the default WebRTC settings:

open RTOpenAI.Api

let conn = Connection.create()
do! Connection.connect ephemeralKey conn

If you need a custom signaling endpoint or a small amount of WebRTC tuning, use Connection.createWithConfig and Connection.connectTo:

open System.Net
open RTOpenAI.Api
open RTOpenAI.WebRTC

let conn =
    Connection.createWithConfig
        { WebRtcClientConfig.Default with
            BindAddress = Some(IPAddress.Parse("127.0.0.1"))
            IceServerUrls = []
            IceGatherTimeoutMs = 250 }

do! Connection.connectTo "http://127.0.0.1:5178/api/realtime/offer" sessionId conn

BindAddress is currently best-effort:

• on Windows/SIPSorcery it is applied to the peer connection
• on iOS and Android it is ignored and logged as a warning

CuaSample

A computer-use-agent (CUA) sample for driving mobile embedded browser based website to accomplish a goal, e.g. extract some information from the site.

Other Projects

In addition to the three projects mentioned above, the following projects are also included in this repo:

• SwiplcsCore: Revamped C# bindings for SWI-Prolog to run under dotnet core (the original bindings are for .Net Framework)

Build and Run Notes

• System requirements:
  • .Net 10 SDK
  • [sudo] dotnet workload install maui
  • XCode and related (matching Maui workload) required on MacOS to target IOS and MacCatalyst app builds
  • Android Studio for Android app target
  • OpenAI API Key
  • Anthropic API Key (for RT.Assistant sample)
  • F# Maui projects cannot be debugged currently with VS Code. You will need either Visual Studio or JetBrains Rider. Both have free editions. For MacOs, Rider is the only viable option. However there are VS Code 'tasks' that can launch MacOS MacCatalyst versions of the included samples.

Important Note:

You will have to update the <TargetPlatforms>...</TargetPlatforms> in the Maui .fsproj files, depending on which platform (MacOs,Windows) your are building on and what you are targeting (IOS,MacCatalyst,Android,Windows):

• RTOpenAI.Api.fsproj
• RTOpenAI.Sample.fsproj
• RT.Assistant.fsproj
• FSPlay

For example, if you are on MacOs then set <TargetPlatforms> to:

• net10.0-ios;net10.0-maccatalyst
• On Windows use net10.0-windows10.0.19041.0
• Add net10.0-android to the list if you have Android Studio or Rider with the Android plugin installed.
• The default is net10.0-ios;net10.0-maccatalyst;net10.0-android.
• For convenience, commented-out versions of the <TargetFrameworks> are included in the .fsproj files.

• Build twice if necessary. Sometimes Maui projects have to be built twice for build errors to go away
• Note that dotnet build at the solution root level is not likely to succeed as there are too many variations possible. Instead set the <TargetFrameworks> and use targeted builds in Rider or Visual Studio.

Package Build

The release builds of Maui apps (not libraries) can take a while so a separate solution is provided RT_FS_NugetPackages.sln to build nuget packages. Use the following command to build packages:

dotnet pack -c Release RT_FS_NugetPackages.sln

Acknowledgements

• Tim Lariviere and others for Fabulous Maui
• WebRTCme - provided the base bindings for Maui WebRTC. These where modified (significantly for IOS) to make them work for RTOpenAI.
• Aaron Clauson and team for SipSorcery - WebRTC for the Windows platform.
• Tau Prolog - a javascript Prolog interpreter used in the RT.Assistant sample.
• Loïc Denuzière and others for FSharp.SystemTextJson - for F# types definitions and the accompanying serialization/de-serialization logic needed to handle OpenAI realtime protocol messages in a strongly-typed way.
• SWI-Prolog team for the base Prolog implementation that was instrumental in developing the Prolog-RAG approach used in RT.Assistant sample.
• Microsoft and .Net Foundation - for dotnet, F#, Maui and Microsoft.Extensions.AI