[WIP] Fix quantization for adapter v2

[rasbt]

Arg, I noticed that llm.int8() quantization breaks for adapter-v2-finetuned models, giving a

    F.linear(input, self.weight, self.bias) + self.adapter_bias
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::BFloat16 != signed char

error.

I tried manually setting the dtype in the new v2 parameters (see this PR) but still get the same issue. Is there perhaps something I should be doing with

    with fabric.device:
        torch.set_default_tensor_type(...)

[awaelchli]

@rasbt This might be the fix we were looking for: #323 :)

[rasbt]

Oh yes, this may be it! Will test it out and continue the discussion in the other PR!

[rasbt]

We can probably close this because of #323