Pinned Loading

1. Transformer-Pretraining-and-Interp Transformer-Pretraining-and-Interp Public

   I Pretrained a Transformer of 134M params from Scratch on around 1.2 Billion Tokens. Now, I am Applying Interpretability techniques to Understand Internals.

   Python

2. GPT-Pretraining GPT-Pretraining Public

   A 134M-parameter GPT written from scratch in PyTorch, no nn.Transformer and no HuggingFace, pretrained on a 7B tokens corpus of FineWeb-Edu, where it consumed 1.2B tokens on one free Kaggle P100. H…

   Jupyter Notebook 6 3

3. Coding-Transformers-using-pytorch Coding-Transformers-using-pytorch Public

   Breaking down the Attention Is All You Need paper into clean, modular PyTorch code — making the Transformer architecture accessible to learners and researchers. Full Article: https://medium.com/@Um…

   Jupyter Notebook