The code is inconsistent with the pseudocode in the paper.

[lartpang]

xcit/semantic_segmentation/backbone/xcit.py

Line 256 in 82f5291

attn = (q @ k.transpose(-2, -1)) * self.temperature

In paper https://arxiv.org/pdf/2106.09681.pdf:

@aelnouby @tanujdhiman