[Feature] Add Local WhisperKit as ASR

[MasamiYui]

Background

The current project uses cloud-based speech recognition services (Alibaba Cloud Tingwu, Volcengine), which presents the following issues:

• Requires uploading audio to the cloud, posing privacy risks
• Depends on network connectivity, cannot be used offline
• Incurs continuous API call costs
• Response latency affected by network conditions

Goals

Replace cloud-based ASR providers with local WhisperKit to achieve fully offline speech recognition capabilities.

Technical Solution 1. Architecture Changes

Current Cloud Provider Flow:

Recording → Upload Audio to OSS → Cloud 
ASR → Download Results → Result Fusion

Local Whisper Flow:

Recording → Local Audio Preprocessing → 
WhisperKit Inference → Result Fusion

2. Core Components

• LocalWhisperService : Implements TranscriptionService protocol, providing a unified transcription interface
• WhisperModelManager : Manages model download, loading, caching, and lifecycle
• LocalWhisperParser : Parses WhisperKit transcription results
• MeetingPipelineManager : Supports local execution path, skipping OSS upload step 3. Model Management
• Download Whisper models from Hugging Face (supports HF Mirror for acceleration)
• Model state machine: Cold (not downloaded) → Warm (downloaded) → Hot (loaded)
• Automatic caching and expiration management
• Offline mode support ( HF_HUB_OFFLINE ) 4. Audio Processing
• Automatic resampling to 16kHz
• Mono channel conversion
• Audio chunking for long audio support

Benefits

1. Privacy Protection : Audio data processed entirely locally, no cloud upload
2. Offline Availability : Works without network connectivity
3. Cost Reduction : No API call fees
4. Faster Response : Local inference without network latency
5. Greater Control : Customizable models and inference parameters

Implementation Plan

• Integrate WhisperKit framework
• Implement LocalWhisperService
• Implement WhisperModelManager
• Update MeetingPipelineManager to support local path
• Add model download and management UI
• Implement result fusion algorithm
• Fix model download timeout issues
• Performance optimization and testing
• Update user documentation
• Gradually remove cloud provider dependencies

Open Issues

1. Model Download : Current tests show model download timeout issues, need to optimize download strategy
2. Performance Optimization : Long audio inference performance needs further optimization
3. Model Selection : Need to determine default model size (base/small/medium)
4. Compatibility : Ensure compatibility with existing features (speaker diarization, AI analysis)

References

• Local Whisper Integration Documentation
• WhisperKit: https://github.com/argmaxinc/WhisperKit
• Hugging Face Models: https://huggingface.co/argmaxinc/whisperkit-coreml

[m13v]

We migrated from cloud ASR to local WhisperKit and can share some practical notes.

On the model state machine (Cold/Warm/Hot) - this is the right approach. One addition: track a "Loading" state between Warm and Hot because model loading takes 2-5 seconds depending on model size and whether the ANE compilation cache exists. On first load after install, CoreML compiles the model for the Neural Engine which can take 30-60 seconds. After that it is cached and loads in 2-5s.

For model selection, we settled on whisper-large-v3-turbo as the default. It is 800MB but the accuracy difference vs base/small is significant for non-English and noisy audio. If you want to support older devices or minimize download size, whisper-small (240MB) is a good fallback with reasonable quality.

The download timeout issue you mentioned - we hit this too. The fix was to use a background URLSession with isDiscretionary = false and implement proper resume support. WhisperKit's WhisperKit.download() method handles this internally if you use the built-in model manager, but if you are downloading from Hugging Face directly, you need to handle partial downloads and resume yourself.

For the audio preprocessing: WhisperKit expects 16kHz mono Float32 audio. If your existing recording pipeline outputs a different format, use AVAudioConverter rather than manual resampling. It handles edge cases (non-integer sample rate ratios, channel mapping) correctly.

For long audio, chunk at silence boundaries rather than fixed intervals. Use AVAudioFile to read the waveform, find low-energy regions (simple RMS threshold), and split there. This avoids cutting words in half which badly hurts transcription quality at chunk boundaries.

[m13v]

Our local WhisperKit transcription service with model management, chunking, and the full pipeline: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

Fazm is an open-source macOS AI agent that uses WhisperKit for all voice transcription, fully local: https://fazm.ai/gh

[MasamiYui]

Our local WhisperKit transcription service with model management, chunking, and the full pipeline: https://github.com/m13v/fazm/blob/main/Desktop/Sources/TranscriptionService.swift

Fazm is an open-source macOS AI agent that uses WhisperKit for all voice transcription, fully local: https://fazm.ai/gh

Thank you for sharing your experience and recommendations. I will give them a try later!

[m13v]

yeah that TranscriptionService file has the full pipeline if you want to see how it all fits together. the trickiest part was the model download/management - WhisperKit models are pretty large and you need to handle the case where the user starts dictating before the model finishes downloading. we show a progress indicator and fall back to Apple's built-in speech recognition until the model is ready