Request for Feedback on Parallel I/O Performance with PnetCDF-Python on HPC Cluster

[wkliao]

On Apr 8, 2025, at 3:59 AM, Sumanth Gopalagowda wrote:

Dear Professor Liao,

I hope this message finds you well.

My name is Sumanth Gopalagowda, ...
As part of my academic project, I am working on computing various deformation measures from snapshots of atomistic simulations.

After computing these measures, I am attempting to write the resulting data to disk in parallel using the PnetCDF-Python interface. However, I am facing performance issues during the I/O stage on our HPC cluster. For instance, writing approximately 72 GB of data takes around 269 seconds on a single node, but the time increases to about 337 seconds when using two nodes. This I/O bottleneck is negating the computational speed-up achieved through parallelization.

I have attached the relevant portion of my code that performs the file writing. I would be extremely grateful if you could take a moment to review it and provide any suggestions or insights on how I might improve its scalability and performance.

Thank you very much for your time and consideration.

Best regards,
Sumanth Gopalagowda

write_out.py.txt

Please let us know what file system you are using (is it a parallel file system?).
If you can edit your short program to add the main function so we can reproduce
the performance number, that will be helpful.

[Sumanthbg]

Hello,
I am using the Panasas file system (panfs). Also, I am adding edited code to reproduce.

write_out.py.txt

[wkliao]

I was able to run the test program. Thanks.

Since I do not have an access to a Panasas, if you can test a few changes
to the codes and let me know if they make a difference, that will be great.
First, can you comment out those romio hints? The data partitioning pattern
used in your test program needs none of these hints.

[Sumanthbg]

Hi, I tried that, and it didn't change the time taken.

[wkliao]

I have a few suggestions.

1. For Panasas, you may need to consult the system administrator, as there may be some
   command-line flags required for using MPI-IO. At the moment, you can add prefix "ufs:" to
   the output file name, i.e. file_save = "ufs:test_output_parallel". This bypasses the
   Panasas specific settings and use generic Unix File System setting in MPI-IO.
2. In order to see whether the problem lies in PnetCDF or MPI-IO, can you try a
   pure MPI-IO test program, such as coll_perf.c
3. Here is just an observation. Is it possible to swap the dimensions of "OutArray", i.e. to

       OutArray = np.zeros((10, atoms_per_rank))
   

   This is because numpy arrays are in row-major. Your test program uses "OutArray[:,j]"
   which results in a noncontiguous I/O buffer.

[Sumanthbg]

I tested the first 2 points,

1. Adding "ufs" to file save did not change the timings.
2. Find the benchmarks as below:
   data file of size 57GB,
   On 2 nodes:
   Collective write time = 351.039425 sec, Collective write bandwidth = 164.084134 Mbytes/sec
   Collective read time = 229.400698 sec, Collective read bandwidth = 251.089036 Mbytes/sec
   On 1 node:
   Collective write time = 185.402592 sec, Collective write bandwidth = 310.675268 Mbytes/sec
   Collective read time = 73.226014 sec, Collective read bandwidth = 786.605696 Mbytes/sec

[wkliao]

Thanks. These timing results show MPI I/O does not take effect as it supposes to.
Most likely, it is an issue with the Panasas file system setting. Please reach out to
the system administrator. Maybe you need to set some environment variables.

I wonder if you can run coll_perf.c with more MPI processes and on more compute nodes.
If the timings fail to reduce, then it is an indicator that the issue has something to do
with the file system.

[wkliao]

FYI. There are few MPI-IO hints for Panasas.
See pages 6 and 7 in ROMIO user guide
There is an example in page 7. You can try to adjust to see if they make any difference.

[Sumanthbg]

Thanks, I am in contact with the system admins. I will let you know if there are any updates.

[Sumanthbg]

Hello @wkliao , I was finally able to get access to the Lustre parallel file system, and below are the time measurements for file output operations (File size 70GB).
Multi-node runs (each node using 40 cores):

Nodes	Writeout Time (s)
1	51.20
2	46.02
4	36.71

Single-node runs with varying core counts:.

Cores	Time for File Output (s)
2	125.11
10	80.75
20	54.25
30	51.51
40	50.65
50	54.76
104	114.62

For the single-node configuration, I observe that after ~20 cores, there are no significant gains in write performance. In fact, performance slightly degrades beyond 40 cores.
I am currently using the following MPI I/O hints (for single node):

info = MPI.Info.Create()
info.Set("romio_cb_write", "enable")
info.Set("romio_cb_read", "enable")
info.Set("romio_no_indep_rw", "true") 
info.Set("cb_buffer_size", "67108864") 
info.Set("cb_nodes", "4") 
info.Set("nc_var_align_size", "1048576") 

Is this behavior expected on Lustre? Or are there additional optimizations or tuning parameters you would recommend—either at the MPI level or on the Lustre side—to improve file output scalability?

[wkliao]

To understand the performance, please consult the system folks for the
highest measurable I/O bandwidth of the Lustre system you are using.
This number varies from system to system.

What is the Lustre file striping configuration you are using?
You can run command lfs getstripe filename to obtain that information.
Increasing striping count usually improves performance.

[Sumanthbg]

I will check with the admins for I/O bandwidth,
below is the striping config:

 lcm_layout_gen:    12
 lcm_mirror_count:  1
 lcm_entry_count:   1
   lcme_id:             131073
   lcme_mirror_id:      2
   lcme_flags:          init
   lcme_extent.e_start: 0
   lcme_extent.e_end:   EOF
     lmm_stripe_count:  16
     lmm_stripe_size:   1048576
     lmm_pattern:       raid0
     lmm_layout_gen:    0
     lmm_stripe_offset: 75
     lmm_pool:          ddn_hdd

[wkliao]

lmm_stripe_count: 16

You are using striping count of 16, which is good for I/O performance.

I observe that after ~20 cores, there are no significant gains in write performance.

I think it is because the maximum communication hardware bandwidth on a single node has been reached.
What is the number of physical cores available on a compute node in your system? Is it 104?

You can try running more nodes to see if the timing continues to decrease.

[Sumanthbg]

Yes, it is 104 cores. 52 cores each in 2 sockets. I will test with more nodes but by using 20-30 cores in each core