nvlink partitioning: improve default partition handling

[tmcroberts97]

The current behaviour for default partitions* is if the tenant attaches a GPU to a partition, and the GPU is already in the default partition, the GPU gets removed from the default partition and added to the new one. However, when the tenant detaches the GPU, or releases the instance, the GPU doesn't get added back into the default partition. Also, tenants currently have no visibility of whether their GPUs are in the default partition.

We should improve this behaviour by:

1. showing that a GPU is in the default partition in the instance status.
2. add a "preserve_default_partitions" flag to the site config. If this flag is true, GPUs will be added back to the default partition when they are detached from user partitions. If false, the GPUs are removed as soon as an instance is created, and added back to the default partition when the instance is removed.

The reason for the flag is that some tenants are making use of default partitions currently because they can use nvlink without any configuration, but I think that's something we want to move away from in the future; there are potential security issues, e.g. if instance GPUs stay in the default partition, they would have shared GPU access with instances owned by different tenants because the other instance is also in the default partition. We can give tenants the same "zero config" benefit by having them use the VPC default logical partition.

*we are referring to NMX-C default partitions here (NMX-C optionally creates a partition and puts every GPU in the rack into it), as opposed to the VPC default logical partition.

[setaati1]

What partition are the machines in Ready in? default partition? an admin partition?
What about a machine that is "out for repair"?
Wouldn't a tray that is released from a Tenant X partition go through scout / cleanup operation, like delete an instance?

[tmcroberts97]

• for machines in Ready state, it depends whether the tenant has added the GPUs to a partition when there was an instance on it previously. If that is the case, the GPUs will be in no partition. Otherwise, Ready machine GPUs should be in the default partition.
• we are still thinking about how we approach OFR partitions.
• yes, in that case we remove the GPUs from the tenant partition, and add them back to the default partition (once we implement the above, currently they wouldn't be in any partition after the instance is deleted).

[ajf]

but I think that's something we want to move away from in the future; there are potential security issues, e.g. if instance GPUs stay in the default partition, they would have shared GPU access with instances owned by different tenants because the other instance is also in the default partition.

Yes, 100%. I'm surprised this didn't come up in the design review? (or maybe it did and there are good reasons to do this), but looking at it now, I don't see any reason a GPU should ever be in a default partition in a multi-tenant partitioned NVLink domain.

[deepak-poornachandra]

Priority changed to P1.