Different stats output with different thread number allocation

[pavgreen9]

Question/Support Request

I was running fastsurfer on a single participant across different thread counts in docker, however the result was unexpected

Depending on the CPU allocation, the time taken to process the image and the stats output was different. This applied across different machines and the same machine.

On our current main machine when we specify 16 threads (which makes use of the whole 32 threads during parallel processing), the time taken to preprocess the image takes 0.26 hours. Surprisingly, however, when using 4 and 8 threads, the time taken was about the same - 0.25 to 0.26 hours. However, once we over allocated our available threads by double (32), the processing time took 1.2 hours. Slightly confused as to why this would happen as it is expected that fastsurfer would allocate the maximum available threads only. Similarly, on our past machine, when we specify 4 threads (on an 8 thread machine), the time taken to preprocess the image is roughly 40-50 minutes. While allocating 8 threads (which overallocates 16 during parallel processing) increases the processing time to 1.5 hours.

Our main concern is with regards to Fastsurfer stats output. The output across different threads were different in the volume segmentation, and surface area and thickness calculation. In terms of volume segmentation, differences were around a few hundred mm^3 depending on the thread count used on the same machine and also the same thread count across different machines. The surface area calculations differ by 10-50 mm^2, while thickness would differ by a few 0.x mm depending on the thread count.

The reason I was testing this out was considering whether I should run multiple docker containers at once whilst limiting each container's CPU usage, which would allow us to preprocess multiple participants at once. However, this usually resulted in a (core dumped) error, and considering the differing stats outcome, I'm not sure if this is a good approach.

My question is whether

• This is expected behavior as the output should be the same regardless of thread count allocation
• Whether this difference is huge enough to warrant preprocessing with an "ideal" thread count
• Is an overallocation of threads a big issue to the output? and why does the processing time differ so much
• Are there are other ways of running multiple participants concurrently using docker

Environment

• FastSurfer Version: 2.1.1 on Docker
• OS: Ubuntu 22.04
• CPU: i9 13900k
• GPU: RTX 4090

Execution

docker run --gpus all
-v /home/chan/Documents/MRI/Participants/${participant}:/data
-v /home/chan/Documents/MRI/Output:/output
-v /home/chan/Documents/MRI:/fs_license
--rm --user $(id -u):$(id -g) deepmi/fastsurfer:latest
--fs_license /fs_license/license.txt
--t1 /data/${participant}.nii
--sid ${participant} --sd /output
--parallel
--threads 16
--batch 16
--vox_size 1

Threads tested on 1, 4, 8, 16, 20, 24, and 32

The images we are running on are anisotropic images (5x.5x1.0) which are downsamples to 1mm3 using fastsurfer's vox_size flag.

[m-reuter]

Hi, first I wonder, do you already see difference after the deep-learning segmentation step (the aseg+DKT.stats and the aparc.DKTatlas+aseg.deep.mgz ) ?

Also you are passing --gpus all, so are you running the segmentation on the GPU ? Probably, as your system has the RTX, but make sure that it says GPU in the log files. --threats should not have an influence on the GPUs (right @dkuegler ?).

For the surface stream (which also edits the segmentation using the surfaces), I would expect differences. The Freesurfer tools may not be deterministic and probably depend on the order of things, so if you parallelise this can change things (even, with the same number of threats in subsequent runs on the same machine). It would be interesting to have the percent changes to better judge if this is purely numerical noise or something else that should be looked at.

Since you only have one GPU that is probably why your parallel cases stop (overflow of GPU RAM). One way around it is to only use CPU also for the segmentation. That will be slower, but you may be able to run more cases in parallel to make up for it. On a large cluster with only few GPUs this usually is the optimal approach.

About the slow speed when increasing the threats too much, it would be good to find out what step in the pipeline is mainly responsible for the slow down as it is large. Probably get clogged up and wait all the time. If segmentation runs on the GPU, then it will still be fast, but the surface module is probably the reason.

[m-reuter]

Oh and your batch size is probably too large. Just use the default on the GPU or shrink it if you want multiple to run in parallel.

[pavgreen9]

Hi, thanks for the reply.

Sorry for the confusion, when I said parallel processing, I meant using the --parallel flag in the hemisphere processing. I've noticed that during parallel hemisphere processing, the thread usage would double what was noted in the --thread, which makes sense as the hemispheres are run in parallel. With regards to running multiple participants in parallel (running different docker containers at once), results are inconsistent with most attempts being unsuccessful regardless of batch or thread size. I also managed to run a single participant with a batch size of 52 (maximum) and two participants at once on two separate docker containers successfully with a batch size of 16 (only once success) so I'm unsure if its a batch size problem. I can't seem to replicate the dumped core error or find it in the logs again, maybe it was just due to chance.

With regards to the deep-learning based segmentation, the output files are the same across machines and threads. I've also checked and the log files which note the use of CUDA, so the GPU is utilised. So no worries there.

The parallel hemisphere processing is the point at which the dumped core error occurs, and also where the preprocessing takes the most time.

Specifically the real-time log would be something like this

RUNNING HEMIs in PARALLEL !!!

RunBatchJobs: Logfile: /output/9001/scripts/recon-surf.log
RunBatchJobs: CMDF: /output/9001/scripts/lh.processing.cmdf
RunBatchJobs: CMDF: /output/9001/scripts/rh.processing.cmdf
Waiting for PID 2818 of (2818 2820) to complete...

and stay here whilst using double the thread count specified by the --thread flag. Issues would occur when the process ends prematurely after this part and doesn't continue with the surface creation. This ending prematurely would result in recon-surf_times.yaml, recon-surf.done, pctsurfcon_log.old and pctsurfcon.log being missing from the scripts file.

[m-reuter]

Many of FreeSurfers binaries are single thread and so changing the number of threads may not have a strong effect on large portion of the recon-surf pipeline. This is probably why you don't see much of a difference if you go down from 8 to 4 etc. We usually don't modify the threats and just use --parallel for the hemis, which helps a lot.

Also the differences you see in the surface pipeline depending on the threats, they should be small, so it would be good to see them in %. We definitely plan to investigate this further.

About the core dumps, could be you do not have sufficient CPU RAM to run multiple docker and surface pipelines simultaneously? I do not understand from your description what you did exactly. How many docker images did you run at the same time? With what flags? For GPU keep in mind that the memory is the bottle neck and not many cases can run simultaneously, probably only 1 case. For the surface pipeline, if you use --parallel, it will need twice as much CPU memory, if you run two cases, then this is 4 times as much plus the overhead from the two docker images. So probably good to have 40-60 GB RAM for that (that is a guess and also needs to be tested).

[dkuegler]

On our current main machine when we specify 16 threads (which makes use of the whole 32 threads during parallel processing)

I believe our usage of threads is probably misleading. If the user specifies --threads 16, the script should not launch 32 threads. This behavior is very counter-intuitive. Thank you for reporting this.

Slightly confused as to why this would happen as it is expected that fastsurfer would allocate the maximum available threads only.

I believe this is an issue in our main "run_fastsurfer.sh" script, which always goes for a single thread. Several scripts in fastsurfer already automatically allocate the number of threads to fit your cpu. In any case, we currently do not cap/limit the number to the number of physical threads, so large numbers can (and will) have weird effects and cause bad performance.

--threats should not have an influence on the GPUs (right @dkuegler ?)

Correct!

fast processing / multi-subject

The FastSurfer pipeline has a segmentation stage (which benefits from a GPU, see --seg_only) and a surface stage (which does not use the GPU, see surf_only). These can also be run one after the other. So if you want to parallelize subjects, you can parallelize the surface stage of multiple subejcts, while processing the segmentation stage separately.

[pavgreen9]

Hi, thanks for getting back.

I will send over a % calculation difference based on the thread count on our current machine and last one, as well as between machines by this weekend. Apologies for not clarifying regarding the docker image, I'm currently just running 2 docker images with a CPU ram of 32GB, this is the code

docker run --gpus all \ -v /home/chan/Documents/MRI/Participants/${participant}:/data \ -v /home/chan/Documents/MRI/Output:/output \ -v /home/chan/Documents/MRI:/fs_license \ --rm --user $(id -u):$(id -g) deepmi/fastsurfer:latest \ --fs_license /fs_license/license.txt \ --t1 /data/${participant}.nii \ --sid ${participant} --sd /output \ --parallel \ --threads 8 \ --batch 8 \ --vox_size 1

Have tried it out with a batch size of 1 as well. I managed to dig out the error and its the same throughout

#@# Sphere lh Wed Jul 19 14:05:09 UTC 2023
/output/9001/scripts

mris_sphere -seed 1234 ../surf/lh.inflated ../surf/lh.sphere

setting seed for random number genererator to 1234
version: 7.3.2
available threads: 8
reading original vertex positions...
projecting onto sphere...
surface projected - minimizing metric distortion...
scaling brain by 0.325...
MRISunfold() max_passes = 1 -------
tol=5.0e-01, sigma=0.0, host=81069, nav=1024, nbrs=2, l_area=1.000, l_dist=1.000
using quadratic fit line minimization
complete_dist_mat 0
rms 0
smooth_averages 0
remove_neg 0
ico_order 0
which_surface 0
target_radius 0.000000
nfields 0
scale 1.000000
desired_rms_height -1.000000
momentum 0.900000
nbhd_size 7
max_nbrs 8
niterations 25
nsurfaces 0
SURFACES 3
flags 0 (0)
use curv 0
no sulc 0
no rigid align 0
mris->nsize 2
mris->hemisphere 0
randomSeed 1234

singular matrix in quadratic form
singular matrix in quadratic form
singular matrix in quadratic form
singular matrix in quadratic form
singular matrix in quadratic form
singular matrix in quadratic form
Segmentation fault (core dumped)
Linux 81069df3452f 5.19.0-46-generic #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s 9001 exited with ERRORS at Wed Jul 19 14:05:55 UTC 2023

For more details, see the log file /output/9001/scripts/recon-all.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Command exited with non-zero status 1
@#@FSTIME 2023:07:19:14:05:08 recon-all N 10 e 46.27 S 4.12 U 259.92 P 570% M 249584 F 6 R 3937530 W 0 c 2910 w 7967 I 0 O 904 L 10.36 8.31 7.36
@#@FSLOADPOST 2023:07:19:14:05:55 recon-all N 10 14.00 9.48 7.80

/output/9001/scripts/rh.processing.cmdf

Noted regarding the parallelisation of -seg and -surf stage, will see if the time saved is significant for us!

[m-reuter]

Have you had a look at the inputs lh.inflated and lh.sphere? Do they look OK? mris_sphere is a FreeSurfer command and probably works for most inputs, but maybe these are already broken due to issues earlier in the pipeline.

[pavgreen9]

Hi, sorry for the late reply.

Here are the stat comparisons compiled for one participant's lh.aparc.DKTatlas.mapped.stats across threads 1, 4, 8, 16, 20, 24, 32. This is with the 8 threads as reference.

Here are the raws just in case

There looks to be a few that are northwards of 2% difference

With regards to the error while running multiple docker containers at once, the stats file only contain aseg+DKT.stats, cerebellum.CerebNet.stats, and l or r h.curv.stats. Apologies, neglected to mention this when I mentioned the lack of files in scripts.

The stats for the ?h.curv.stats that does get outputted in parallel dockers are the exact same as their single docker counterpart (assuming same thread number).