llm-pretraining

Here are 3 public repositories matching this topic...

NoCheeseMercy / KitAI

A 197M-param decoder-only LM trained from scratch on FineWeb-Edu — PyTorch with RoPE, GQA, SwiGLU, 32k BPE, built for 6GB VRAM

nlp cuda pytorch transformer language-model rope gqa swiglu bpe-tokenizer small-language-model fineweb llm-pretraining

Updated Sep 5, 2026
Python

SakethBettakonda / LLM-Pretraining-Project

Star

Implementation of ELECTRA pretraining architecture on a logic-focused corpus. Achieved 96.42% discriminator accuracy with 80% reduction in discriminator loss. Includes generator-discriminator training pipeline, domain adaptive pretraining, and comparison with RoBERTa baseline

nlp transformers pytorch electra llm-pretraining

Updated Jul 17, 2026
Jupyter Notebook

chiangjw90 / gravity-labeler

Star

Architecture-agnostic data profiling via field physics — improves LLM pretraining by 1.0–1.9% on Transformers

machine-learning neural-network data-curation curriculum-learning pretraining data-centric-ai large-language-models llm-pretraining physics-inspired-ai

Updated May 22, 2026
Python

Add this topic to your repo

To associate your repository with the llm-pretraining topic, visit your repo's landing page and select "manage topics."

Learn more