Skip to content

Tansformer setup question #9

Description

@suirenjiruka

In the SnapMoGen/tree/main/model/transformer/transformer.py

126 self.lvl_embed = nn.Embedding(len(self.patch_sizes), self.latent_dim)
127 nn.init.trunc_normal_(self.lvl_embed.weight.data, mean=0, std=init_std)

143 self.apply(self.__init_weights)

I found that self.lvl_embed get reset by self.apply(self._init_weights) again.
I am not why the nn.init.trunc_normal
should be implemented before _iniy_weights.
In my understanding, the weight initialization would be overwritten in line 143.

I doubt it might a small setup mistake.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions