Guide · Open source

How to use train-llm-from-scratch: setup, training and tips

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

train-llm-from-scratch is an MIT-licensed tutorial repository that builds a transformer language model in plain Python and PyTorch and walks through training it. It is written for students, developers with PyTorch experience and researchers who want to see how each piece works. After this guide you will know how to install the repo, where the main stages live, and how to run a small model on a CPU or a larger one on a GPU.

train-llm-from-scratchFareedKhan-dev/train-llm-from-scratch · ★ 13.1k · MIT Install options & downloads

Quick start

1Clone the repository. Run git clone on the GitHub URL to download the code to your machine.
2Enter the folder. Change into the train-llm-from-scratch directory with cd.
3Install in editable mode. Run pip install -e . so the config, src, data_loader and ui packages are importable.
4Pick a stage to run. Start with data preparation and pretraining, or use the smoke configs in configs/smoke/ for a quick check.
5Adjust settings. Edit the JSON file for the stage or pass overrides such as --lr on the command line.

Who should use train-llm-from-scratch?

The README describes three audiences: students who read the material top to bottom, developers who want readable reference code they can copy and run, and researchers interested in how SFT, reward models, DPO and GRPO are implemented from scratch.

It is a learning path, not a production library. The author states that the code avoids trl, peft and transformers on purpose, so you read every algorithm directly.

Students learning how a transformer is built and trained step by step
Developers with Python and PyTorch experience who want readable reference code
Researchers studying post-training methods implemented by hand
People without a GPU who want a small CPU training example

What you need before you install

The README lists these prerequisites: basic object oriented programming, neural networks and PyTorch. Python 3.10 or newer is required, and Git is needed to clone the repository. uv is optional.

You do not need a GPU to start. The README says the laptop track trains a small model on a CPU in minutes. For bigger models a GPU is needed, and a free Colab or Kaggle T4 is enough for the 13 million parameter model.

Python 3.10+
PyTorch
Git
uv (optional)

Install train-llm-from-scratch from source

The project is installed by cloning the repository and running an editable install. The editable install puts config, src, data_loader and ui on your import path, so you do not need to set PYTHONPATH by hand.

If you use uv, the README says uv sync does the same and also installs the test tools. On a Linux machine without an NVIDIA GPU, install the CPU build of PyTorch first, because the default Linux wheel bundles CUDA.

Optional extras: .[train] for datasets and wandb, .[ui] for the Streamlit control panel, .[docs] for the documentation site, .[all] for everything

git clone https://github.com/FareedKhan-dev/train-llm-from-scratch.git
cd train-llm-from-scratch
pip install -e .

Understand the two config systems

The repo has two config systems. config/config.py holds plain Python constants for the legacy pretraining script scripts/train_transformer.py. config/post_training_config.py together with JSON files in configs/ drive the other stages, including pretraining the bigger base, SFT, reward, DPO, PPO and GRPO.

For the newer stages you edit a small JSON file per stage. Any field can also be overridden on the command line, for example --lr 2e-5 --batch_size 16.

config/config.py: legacy pretraining constants
config/post_training_config.py and configs/: all other stages
Command line overrides such as --lr and --batch_size

--lr 2e-5 --batch_size 16

Run the pipeline stages in order

The README walks the full path: raw text becomes tokens, tokens feed a Transformer, next-token loss trains a base model, then SFT, a reward model, PPO or DPO, GRPO, and finally evaluation and chat.

Start with data preparation and pretraining. Post-training stages build on a base model, so run them after pretraining. The README also provides a tiny configs/smoke/ variant of each stage for fast checks.

Step 1: Preparing the data
Step 2: Building the model from small pieces
Step 3: Pretraining the base model
Step 4: Generating text
Step 5: Post-training (SFT, reward model, DPO/ORPO/KTO, PPO, GRPO)
Step 6: Evaluation
Step 7: Talking to the model

Choose hardware and model size

The README gives a GPU table listing memory and practical maximum training size. For example, an NVIDIA RTX 4060 with 8 GB is listed for about 1B parameters, and a Tesla T4 with 16 GB for about 1.5B to 2B. These are the README's estimates, so check the table before you commit to a large config.

If a large config runs out of memory, the pretraining script has opt-in flags: --amp, --grad-checkpointing and --grad-accum.

--amp
--grad-checkpointing
--grad-accum

What people use it for

Learning how tokens, attention and the transformer block fit together by reading the code
Training a small model on a laptop CPU to see the loss fall and sample text
Studying how SFT, reward models, DPO and GRPO are written by hand in PyTorch
Comparing the original model with the modern variant that uses rotary positions and RMSNorm

Frequently asked questions

Is train-llm-from-scratch free to use?

Yes. The repository is licensed under MIT, as listed on its page. You can read, run and modify the code under those terms.

What are the requirements for train-llm-from-scratch?

The README lists Python 3.10 or newer, PyTorch, Git, and uv as optional. It also expects basic object oriented programming, neural network and PyTorch knowledge.

Can I train train-llm-from-scratch without a GPU?

Yes. The README says a small model can be trained on a laptop CPU, and the laptop track does this in minutes. On Linux without an NVIDIA GPU, install the CPU build of PyTorch first.

How do I install train-llm-from-scratch?

Clone the repo with git clone, enter the folder, and run pip install -e . The README also notes that uv sync performs the same setup and installs the test tools.

Is train-llm-from-scratch good for production use?

No. The README presents it as a learning path and says it is not a production-ready LLM or chat product. Use it to understand the method rather than to ship a model.

What are alternatives to train-llm-from-scratch?

Comparable projects listed for this item are nanoGPT, minGPT and TRL. The repo differs from TRL because it avoids trl, peft and transformers on purpose and implements the algorithms by hand.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use train-llm-from-scratch if you want to read and run a transformer and its post-training methods written in plain PyTorch, and you already know Python and PyTorch. Pick a standard library such as TRL or a production framework if you need to fine-tune an existing pretrained model or ship a chat product.

Install options & downloads