A Session that contains a Gemm or Conv operation only works at the size it was constructed with. Calling infer with a different size crashes with std::out_of_range from layoutStore.at. I came across this issue while adding dynamic input shape support to the alpaka GPU backend (ML4EP/SOFIE#45). With symbolic dims like a batch size N, the whole point is to build the Session once and run infer at different sizes which can't be done as of now
cuBLASLt layouts are created up front by addLayoutConfig, which the generated Session constructor calls with the constructor's shape values. At inference time executeMatmul and getOrComputeAlgo look layouts up with layoutStore.at(...), keyed by the runtime (rows, cols) of the multiply. For a static model the two always match. For a dynamic model the runtime dims come from the infer arguments, so any size not seen at construction has no entry in the store and .at throws.
A Session that contains a Gemm or Conv operation only works at the size it was constructed with. Calling infer with a different size crashes with std::out_of_range from layoutStore.at. I came across this issue while adding dynamic input shape support to the alpaka GPU backend (ML4EP/SOFIE#45). With symbolic dims like a batch size N, the whole point is to build the Session once and run infer at different sizes which can't be done as of now
cuBLASLt layouts are created up front by addLayoutConfig, which the generated Session constructor calls with the constructor's shape values. At inference time executeMatmul and getOrComputeAlgo look layouts up with layoutStore.at(...), keyed by the runtime (rows, cols) of the multiply. For a static model the two always match. For a dynamic model the runtime dims come from the infer arguments, so any size not seen at construction has no entry in the store and .at throws.