(venv) [xinjingjing@dev-gpu-node-09 InstructGLM]$ python train_lora.py \
> --dataset_path data/belle \
> --lora_rank 8 \
> --per_device_train_batch_size 2 \
> --gradient_accumulation_steps 1 \
> --max_steps 52000 \
> --save_steps 1000 \
> --save_total_limit 2 \
> --learning_rate 2e-5 \
> --fp16 \
> --remove_unused_columns false \
> --logging_steps 50 \
> --output_dir output
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA SETUP: CUDA runtime path found: /usr/local/cuda-11.2/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 112
CUDA SETUP: Loading binary /data/chat/InstructGLM/venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda112.so...
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Overriding torch_dtype=None with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:12<00:00, 1.51s/it]
Traceback (most recent call last):
File "/data/chat/InstructGLM/train_lora.py", line 170, in <module>
main()
File "/data/chat/InstructGLM/train_lora.py", line 128, in main
model.gradient_checkpointing_enable()
File "/data/chat/InstructGLM/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1584, in gradient_checkpointing_enable
raise ValueError(f"{self.__class__.__name__} does not support gradient checkpointing.")
ValueError: ChatGLMForConditionalGeneration does not support gradient checkpointing.
训练的时候提示如下错误：