High memory usage when processing large input data

[Guanhua-L]

Hello,
I am using your code, but when handling large input data, the memory usage becomes excessively high, causing the program to crash.

The input data size I am using is:
X_test_pd.csv: 67,907
X_tr_pd.csv: 270,522
Y_test_pd.csv: 67,907
Y_tr_pd.csv: 270,522

Here is some system information:
OS: Ubuntu 20.04.2 LTS
CPU: Intel Xeon E5-2620 v4 @ 2.10GHz
RAM: 188GB
GPU: NVIDIA GeForce GTX 1080 Ti (11GB VRAM)
CUDA Version: 11.7 (V11.7.64)
Python Version: 3.8.20