fineweb

LiteGPT: A 124M Small Language Model (SLM) pre-trained on FineWeb and fine-tuned on Alpaca.

Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.

A 134M-parameter GPT written from scratch in PyTorch, no nn.Transformer and no HuggingFace, pretrained on a 7B tokens corpus of FineWeb-Edu, where it consumed 1.2B tokens on one free Kaggle P100. Held-out perplexity 38.89 against a Chinchilla prediction of 38.70.

Exploring the BBC News subset of the FineWeb dataset (via HuggingFaceFW/fineweb's dated subsets on HF), originally a Common Crawl dataset.

A private index of the public web — backfill history + live-tail RSS/Common Crawl/GDELT onto your laptop, query it like a table. One Python process, fully local.

Training GPT-2 on FineWeb-Edu in JAX/Flax

A 197M-param decoder-only LM trained from scratch on FineWeb-Edu — PyTorch with RoPE, GQA, SwiGLU, 32k BPE, built for 6GB VRAM

A 66M parameter decoder-only transformer language model implemented from scratch in PyTorch. Features a custom SentencePiece tokenizer, RoPE positional embeddings, SwiGLU feed-forward network, per-layer KV cache for efficient autoregressive inference, and a Svelte-based streaming chat interface.

OpenAI Parameter Golf competition entry — adaptive Hessian-sensitivity GPTQ clipping on PR #1855's stack. 1.06310 BPB (3-seed mean), PR #1962.

A minimal GPT model pretraining pipeline (mostly taken from Andrej Karpathy's build-nanogpt repo)

Do quality filters pull benchmark questions into pretraining corpora? Injects MMLU, GSM8K, GPQA, ARC, HellaSwag, PIQA and TruthfulQA items into a corpus and ranks them with six quality classifiers, including DCLM, FineWeb-Edu and Gaperon.

Decoder-only LLM from scratch with reproducible data pipelines, tokenizer/sharding workflows, and GPU training.

FineWeb-Edu dataset analysis using Apache Spark - DSC 232R group project