Initial dynamic input support for gpu codegen#45
Draft
harz05 wants to merge 13 commits into
Draft
Conversation
2946253 to
78eb28e
Compare
01ddd4c to
e8623a1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds dynamic shape (symbolic batch dimension N) input support to the gpu code generator.
While generating a dynamic model on the GPU path it failed to compile. The dynamic intermediate tensor buffers were emitted with a type alias built on the runtime device object
devAccwhere a type is needed, declared under the namebufDev_<name>while every operator referencesdeviceBuf_<name>(so the buffer was undefined at use), and allocated into a discardedautolocal in the constructor with a hardcoded float type, so the member was never set. After fixing those, a second error showed up becausealpaka::Bufhas no default constructor, so the bare member declaration would not compile. The fix declares the dynamic buffers as Session members with the correct Buf type anddeviceBuf_name according to the tensor dtype, default initializes each member to a one element placeholder buffer to satisfy the no default constructor requirement, and allocates the real buffer in the constructor sized by the runtime length once N is known.Modified operators that now support dynamic input-
Kernel argument convention
Index math kernels that reference a runtime dimension now receive the model shape parameters (for example N) as
size_tkernel arguments, supplied at thecreateTaskKernelcall site. This matches the existing ScatterElements kernel. Flagging it for review, if there could be some different methods to do it instead of thisAdded gtests(
DynamicTileandDynamicEqual) to verify the changes made. For this test two different batch sizes of N = 1 and N = 8 were chosen and the output was compared against an independent host reference.Tests verified on NVIDIA T4.