The CLIP loss implementation seems not completely right

[HobbitLong]

Hi,

Thank you for sharing your code!

If I am not missing something, I think the CLIP loss implementation here is not completely correct, in the sense that one should gather with gradient. Do you have a specific reason for not using it (since you have already implemented it for SimCLR), or you have back-propagated the gradient somewhere that I have missed?