Hello,
I tried to train the model with intensity values but it always slows down after two hours and crashes around epoch 32-37. I now tried to resume training with the latest checkpoint but I get the following error:
Size of train_dataset = 3000
Size of test_dataset = 604
Size of val_dataset = 322
Batch size = 2
[2026-05-06 23:39:54,719][torch_points3d.datasets.base_dataset][INFO] - Available stage selection datasets: ['test', 'val']
[2026-05-06 23:39:54,719][torch_points3d.datasets.base_dataset][INFO] - The models will be selected using the metrics on following dataset: val
[2026-05-06 23:39:56,850][torch_points3d.trainer][INFO] - EPOCH 32 / 150
0%| | 0/1500 [00:12<?, ?it/s]
Traceback (most recent call last):
File "train.py", line 17, in main
trainer.train()
File "/home/lukas/ForAINet/PointCloudSegmentation/torch_points3d/trainer.py", line 157, in train
self._train_epoch(epoch)
File "/home/lukas/ForAINet/PointCloudSegmentation/torch_points3d/trainer.py", line 212, in _train_epoch
self._model.optimize_parameters2(epoch, i, self._dataset.batch_size)
File "/home/lukas/ForAINet/PointCloudSegmentation/torch_points3d/models/base_model.py", line 274, in optimize_parameters2
self._grad_scale.step(self._optimizer) # update parameters
AttributeError: 'NoneType' object has no attribute 'step'
Is there something else except the checkpoint_dir path that should be changed?
Hello,
I tried to train the model with intensity values but it always slows down after two hours and crashes around epoch 32-37. I now tried to resume training with the latest checkpoint but I get the following error:
Size of train_dataset = 3000
Size of test_dataset = 604
Size of val_dataset = 322
Batch size = 2
[2026-05-06 23:39:54,719][torch_points3d.datasets.base_dataset][INFO] - Available stage selection datasets: ['test', 'val']
[2026-05-06 23:39:54,719][torch_points3d.datasets.base_dataset][INFO] - The models will be selected using the metrics on following dataset: val
[2026-05-06 23:39:56,850][torch_points3d.trainer][INFO] - EPOCH 32 / 150
0%| | 0/1500 [00:12<?, ?it/s]
Traceback (most recent call last):
File "train.py", line 17, in main
trainer.train()
File "/home/lukas/ForAINet/PointCloudSegmentation/torch_points3d/trainer.py", line 157, in train
self._train_epoch(epoch)
File "/home/lukas/ForAINet/PointCloudSegmentation/torch_points3d/trainer.py", line 212, in _train_epoch
self._model.optimize_parameters2(epoch, i, self._dataset.batch_size)
File "/home/lukas/ForAINet/PointCloudSegmentation/torch_points3d/models/base_model.py", line 274, in optimize_parameters2
self._grad_scale.step(self._optimizer) # update parameters
AttributeError: 'NoneType' object has no attribute 'step'
Is there something else except the checkpoint_dir path that should be changed?