Improvement ideas

[hansmbakker]

• use github copilot sdk as chatclient?
• Entra Agent ID?
• update module 3 (replace kernel memory)
• update module 4
  • which LLM to use for agents? scoring on task completion
  • more focus on
    • agent skills
    • tools
  • booking skill with some tools
• module 5
  • frontend integration? --> copilotkit?
  • realtime input (conversation? audio? transcription?) with product recommendations?
    • state that is being updated
  • MAI models?

[hansmbakker]

@roycornelissen Related to the realtime speech I found the following resources:

• .NET: Real-time audio processing multi-agent example microsoft/agent-framework#1170 -> needs to be updated to current Agent Framework, needs to be converted to local speech processing
• https://livekit.com/blog/nemotron-3.5-asr-multilingual-teleprompter has a demo on wrapping a model behind an openai-compatible transcriptions endpoint locally
• https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
• https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
• https://huggingface.co/spaces/tvosch/Dutch-ASR-Leaderboard

MAI models seem not available locally; from Microsoft's offering it would then be the Phi series, but the only model from that series that supports audio is the multimodal series which has 10x more parameters (6B instead of 0.6B) and while the Phi-4-multimodal model has an accuracy comparable to parakeet, it is around ~20x slower.

Parakeet: a Google AI query nemotron-3.5-asr-streaming-0.6b vs parakeet-tdt-0.6b-v3 suggests that nemotron-3.5-asr-streaming-0.6b is better for realtime streaming while parakeet v3 has a slightly higher quality at a lower speed.

• https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3 seems an interesting model - it is fast enough to run locally and has support for european languages
• https://github.com/achetronic/parakeet and https://github.com/groxaxo/parakeet-tdt-0.6b-v3-fastapi-openai have a ready available implementation of that model with openai-transcriptions endpoint

Using Foundry Voice Live there is this: https://github.com/microsoft-foundry/voicelive-samples/blob/main/csharp/CustomerServiceBot/AudioProcessor.cs