Shrink/expand questions

[PaulinaEster]

Shrink/expand questions

Hello, I'm trying to test the shrink/expand function of charm++, but I'm not sure if it's working. I followed the tutorial at https://charm.readthedocs.io/en/latest/charm++/manual.html#malleability-shrink-expand-number-of-processors, but when I run the client command (to expand processes), it stops the node and then the application doesn't start it again. Furthermore, it seems the application freezes when it tries to do a new Load Balance after the shrink/expand.

1. Is there another way to test the shrink/expand?
2. What would be the ideal behavior of the jacobi2D example presented in the tutorial?
3. Many articles in the literature cite AMPI as a possible resource malleability tool. Is this at the node level (add or remove a node) or at the process level?
4. Can the shrink/expand function be added to AMPI in the build, or is it only for charm++ for now?

Originally posted by @PaulinaEster in #3914

[HawkmoonEternal]

We have experienced the same problem.

We tested the official Charm++ 8.0.0 release, but the shrink/expand feature did not work.
Then, we switched to the shrinkexpand-fix branch. On this branch, expand, i.e., increasing the number of PEs worked, but we could not get the shrink operation to work.

In a recent publication, the authors state: "We used the non-SMP mpi-linux-arm8 build of Charm++ with shrink/expand enabled, built with OpenMPI v3.1." We haven't tried this version yet; maybe it only works with this particular version.

Any additional information on the current status of the shrink-expand feature would be very much appreciated.

[PaulinaEster]

Hello @HawkmoonEternal , thank you for reporting this. Could you please describe exactly how you tested this functionality, including how many compute nodes were used, how many cores each node had, and which commands were used to build Charm++ and to run the test? By the way, was the test the same Jacobi2D example provided in the shrink/expand tutorial?

[HawkmoonEternal]

Hello @PaulinaEster, we used the following setup:

• Branch: shrinkexpand-fix

• Build:
  ./build charm++ netlrts-linux-x86_64 --enable-shrinkexpand -j4 --force
cd netlrts-linux-x86_64 && make install
export PATH=$CHARM_ROOT/netlrts-linux-x86_64-install/bin:$PATH
cd $CHARM_ROOT/examples/charm++/shrink_expand && make && cd jacobi2d-iter && make

• Minimal Example (Grow 1 -> 2 procs/nodes):

-> create mynodelistfile with the following content (replace with real hostnames):

host [hostname1]
host [hostname2]

In Terminal 1 (server) run:
cd $CHARM_ROOT/examples/charm++/shrink_expand/jacobi2d-iter
charmrun +p1 ./jacobi2d 2000 200 +balancer GreedyLB ++nodelist ./mynodelistfile ++server ++server-port 1234

In Terminal 2 (client) run:
cd $CHARM_ROOT/examples/charm++/shrink_expand
./client n1 1234 1 2

It should also work with more processes and nodes and different problem sizes.
However, shrinking did not work.

I'm adding @tanaytekin to this thread as he did most of the work.

[GuilhermeMVasquez]

Hello @HawkmoonEternal , thank you for supplying your configurations.

I have tested the shrinkexpand-fix branch as mentioned above and can attest that the expand feature does work with some limitations.

Shrinking, as of yet, has also not been working for us.

I would be curious to know if you or @tanaytekin have also encountered problems when starting execution with more than 1 process, as this is where I ran into problems with the expand feature.

I am running a test environment with 3 compute nodes each having 2 processor cores. If I start charm with +p1, it is possible to expand it to any new number of processes to make use of all the cores available in the cluster. The curious thing is when starting charm with +p2 or above. In this scenario, we have observed that we are able to only expand to have one process per node (i.e. 3 procs in my environment). For example, if we try to expand from 2 procs to 4 with ./client master_node port 2 4 then we encounter an error that appears to be related to socket request that ultimately terminates execution.