Error on a slurm cluster: there is insufficient memory for the Java Runtime Environment to continue.

[zhong-yy]

Hi, I recently used python subprocess and multiprocessing packages to run TauP parallelly on a slurm cluster. There are ~256 cores and 1.5T memory available on the cluster. When the number of allocated cpus is larger than 64, there is always a memory error. However, when I checked the memory usage with top, only about 48G was used and there was plenty of free memory. I have searched for as much information related to java and slurm as I can but still haven't found the solution.

Log

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (gcTaskThread.cpp:48), pid=1114048, tid=0x00001491cd1ef700
#
# JRE version:  (8.0_172-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

Setting of my sbatch script

#!/bin/bash
### job name
#SBATCH -J "taup"
#SBATCH --nodes=1
#SBATCH --ntasks=1

#SBATCH --cpus-per-task=100
#SBATCH --mem-per-cpu=5000M

export JAVA_TOOL_OPTIONS=-Xmx400G
export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1
export NUMEXPR_MAX_THREADS=${SLURM_CPUS_PER_TASK}

...

I have tried changing -Xmx using export JAVA_TOOL_OPTIONS=-Xmx128M, but it didn't work.

[crotwell]

What version of TauP are you using?

Are you sure that your python code is not spawning too many instances of TauP or shutting them down incorrectly? In other words, can you reproduce this memory error running TauP directly?

I have done some basic memory profiling and have not been able to get TauP to use more than a few hundred Mb. Possible that there is a memory leak, but without more details on the type of requests you are making and the velocity model it is hard to give much more help.

[zhong-yy]

Hi @crotwell , I am sorry for my late reply.

The version is TauP 2.6.1 and the jdk version is 1.8.0_172. I cannot reproduce this memory error when running TauP directly.

For the moment, I avoid this problem by limiting the number of processes to be less than about 55, but there is a lot of idle computing resources.

To be honest, although I suppose the TauP instances are shut down correctly, I am not sure whether python multiprocessing really does so. My python code is something like

def pred_travel_time(mod,  sta_lat,    sta_lon,    eq_lat,    eq_lon,  eq_dep,    phases,):
    taup_output = subprocess.run(
        [
            "taup",
            "time",
            "-ph",
            ",".join(phases),
            "-mod",
            mod,
            "-h",
            str(eq_dep),
            "-sta",
            str(sta_lat),
            str(sta_lon),
            "-evt",
            str(eq_lat),
            str(eq_lon),
        ],
        capture_output=True,
    )

    # Parse the result
    # remove the first few lines of the output
    taup_output_decoded = (
        taup_output.stdout.decode()
        .rpartition("------------------------------------------------")[-1]
        .strip()
    )
    if taup_output_decoded:
        try:
            result = [x.split() for x in taup_output_decoded.split("\n")]
            p_list = [float(x[3]) for x in result if x[2] in ["p", "P"]]
            s_list = [float(x[3]) for x in result if x[2] in ["s", "S"]]
            p_travel = min(p_list) if len(p_list) > 0 else np.nan
            s_travel = min(s_list) if len(s_list) > 0 else np.nan
        except Exception as e:
            print(result, flush=True)
            raise (e)
    else:
        p_travel = np.nan
        s_travel = np.nan
    return p_travel, s_travel

if __name__ == "__main__":
...
    with ctx.Pool(processes=num_processes, maxtasksperchild=5) as pool:
        tra_times = pool.starmap(pred_travel_time, arg_list)
        print("Finished predicting p and s") 
...