value about Temperature

[coallar]

Hi, I have a problem about: what is the valu of T or self.tau do you have choiced? or How can i set the value while training my project?
T = self.tau
# taken from https://github.com/peterliht/knowledge-distillation-pytorch/blob/master/model/net.py#L100
# with slight modifications
distillation_loss = F.kl_div(
F.log_softmax(outputs_kd / T, dim=1),
F.log_softmax(teacher_outputs / T, dim=1),
reduction='sum',
log_target=True
) * (T * T) / outputs_kd.numel()

[soroush-abbasi]

Hi,

We don't use any teacher or distillation loss during training. We train all models from scratch without any teacher. Please let me know if you have any questions. Thanks!