This is a testing experiemnts & custom blocks added to to Whisper to improve it for a specific competition. I use pre-trained weights even with custom blocks.
The actual attention mechanism uses log-mel spectrograms, and I'm playing with custom blocks for specific frequency bins!
All the credit for baseline/pretrained Whisper to OpenAI. Whisper's code and model weights are released under the MIT License.