Guide · Open source
How to use llama3-from-scratch: a step-by-step walkthrough
Updated · written with AI from the project's own documentation and checked against our catalogue data
llama3-from-scratch is an MIT-licensed Jupyter notebook by naklecha that rebuilds parts of Meta's Llama 3 language model one tensor operation at a time. It loads the official weight file and applies each step by hand, from tokenizing a prompt to computing attention. This guide walks through how to set it up and what each stage of the notebook shows, so you can follow how a text prompt becomes numbers and then gets processed.
llama3 from scratchnaklecha/llama3-from-scratch · ★ 15.2k · MIT Install options & downloads Quick start
- 1Clone the repo. Run the git clone command from the README to get the notebook locally.
- 2Install the prerequisites. Make sure Python 3, Jupyter Notebook, PyTorch, tiktoken and matplotlib are available.
- 3Download the Meta Llama 3 weights. Get the weights from Meta's official Llama downloads page and place them in the folder the notebook expects, Meta-Llama-3-8B.
- 4Open the notebook and run the tokenizer cells. Load the tokenizer and encode a prompt to see how text becomes token IDs.
- 5Load the model file and config. Read consolidated.00.pth and params.json to inspect tensor names and model settings.
- 6Follow the embedding and normalization cells. Check the tensor shapes after each step, as the author recommends.
- 7Work through the first attention head. Reshape the query weights and compute per-token queries for head 0 of layer 0.
What you need before you start
The notebook assumes basic Python and familiarity with tensors and matrix shapes. The README lists Python 3, Jupyter Notebook, PyTorch, tiktoken and matplotlib as prerequisites.
You also need the model weights. The README says you must download them before running the file, and links to Meta's official Llama downloads page. The notebook expects a folder named Meta-Llama-3-8B containing tokenizer.model, consolidated.00.pth and params.json.
- Python 3
- Jupyter Notebook
- PyTorch
- tiktoken
- matplotlib
- Meta Llama 3 weights (downloaded separately)
Clone the repository and open the notebook
Clone the repository using the command from the README. Then open the notebook in Jupyter and run the cells in order, since later cells depend on variables defined earlier.
The README is truncated, so the walkthrough it describes covers only the early stages of the model.
git clone https://github.com/naklecha/llama3-from-scratchTokenize a prompt
The notebook uses tiktoken to load the Llama 3 tokenizer file and encode a prompt into integer token IDs. It prepends a begin-of-text special token (128000) before the encoded prompt. The author states that the BPE tokenizer is intentionally not implemented, and points to Andrej Karpathy's minbpe project for a clean BPE implementation.
Printing the tokens shows each piece of the prompt as its own token, which makes it easy to see how text is split before any model math happens.
tokens = [128000] + tokenizer.encode(prompt)Read the model file and its config
The notebook loads consolidated.00.pth with torch.load and lists the first tensor names, such as tok_embeddings.weight and layers.0.attention.wq.weight. Instead of relying on Meta's model classes, it reads the file one tensor at a time.
params.json holds the model settings. The notebook uses them to get values such as the model dimension, number of layers, number of attention heads, vocabulary size and norm_eps. The author notes the config tells you the model has 32 transformer layers and 32 attention heads per block.
with open("Meta-Llama-3-8B/params.json", "r") as f:
config = json.load(f)Turn tokens into embeddings and normalize them
Each token ID is looked up in an embedding table, producing one vector per token. The author highlights shape tracking: a 17-token prompt becomes a 17 by 4096 tensor. The notebook uses an inbuilt PyTorch embedding layer for this step.
Next comes RMS normalization, implemented in a small rms_norm function. The shapes stay the same; only the values change. norm_eps prevents dividing by zero.
- Embedding output for the example prompt: [17x4096]
- After normalization: still [17x4096]
token_embeddings = rms_norm(token_embeddings_unnormalized, model["layers.0.attention_norm.weight"])Compute query vectors for attention
The notebook loads the attention weight matrices for the first layer. The query, key, value and output matrices are stored together, so the notebook reshapes the query matrix to separate the 32 heads, then takes the first head's query weights.
Multiplying the normalized embeddings by that head's query weights gives one 128-length query per token, a [17x128] tensor. The notebook then turns to positional encoding, since identical words such as the three uses of "the" in the example prompt need different vectors depending on their position.
- Query matrix reshaped to [32x128x4096]
- First head's query weights: [128x4096]
- Per-token queries for the first head: [17x128]
q_per_token = torch.matmul(token_embeddings, q_layer0_head0.T)What people use it for
- Learning how a prompt is split into tokens before the model sees it
- Checking tensor shapes at each stage of a transformer forward pass
- Studying why attention query vectors need positional information
- Reading Llama 3 weight files tensor by tensor to understand their layout
Frequently asked questions
Is llama3-from-scratch free to use?
Yes, the repository is licensed under MIT. The notebook itself is free to clone and read, but you must download the Llama 3 weights separately from Meta's official page.
What do I need to install to run llama3-from-scratch?
The README lists Python 3, Jupyter Notebook, PyTorch, tiktoken and matplotlib as prerequisites. Clone the repository with git and then open the notebook in Jupyter.
Can I use llama3-from-scratch to chat with Llama 3?
No. The project is an educational walkthrough. The README does not describe serving or running the model for chat or apps, so it is not a production inference server.
Does llama3-from-scratch include a full BPE tokenizer?
No. The author says the BPE tokenizer is intentionally not implemented and uses tiktoken instead. For a clean BPE implementation, the README points to karpathy/minbpe.
Why do I need to download the weights first?
The notebook loads tensors directly from Meta's model file, consolidated.00.pth, so the weights must be present before the code runs. The README links to Meta's official download page.
What are good alternatives for learning transformers from code?
The README lists comparable projects: karpathy/nanoGPT, karpathy/minbpe and karpathy/llama2.c. Those are useful if you want a different model or a focus on tokenization.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
llama3-from-scratch suits learners with basic Python and linear algebra who want a tensor-shape-level view of how a transformer processes a prompt. It is not suited to running Llama 3 for chat or building apps, and the truncated README covers only the early steps. If you want a complete implementation or a production server, look elsewhere.
Install options & downloads