llama3 from scratch
A from-scratch walkthrough of Llama 3 inference in a Jupyter notebook, one tensor operation at a time.
- Stars
- 15.2k
- Forks
- 1.3k
- Licence
- MIT
- Language
- Jupyter Notebook
- Last update
- 2.5 years ago
- Type
- Learning resource
No commits in over a year — check for forks or alternatives before adopting.
What it is
This is a Jupyter notebook that rebuilds parts of Meta's Llama 3 language model step by step, loading the official weight file and applying each operation (tokenizing, embedding, normalization, attention) by hand. It is meant to show how a transformer language model turns a text prompt into numbers and then computes with them. The README is truncated, so it covers only the early stages of the walkthrough.
- Follow the notebook to trace how a prompt is split into tokens and mapped to 4096-dimensional embeddings
- Inspect the shapes of Llama 3 attention weights and see how query matrices are split across heads
- Study the RMS normalization formula as implemented in the notebook
Great for
- Learners who understand basic Python and want to see how a transformer works internally
- Engineers who read model code and want a concrete, tensor-shape-level explanation
- Students studying attention and positional encoding
Not ideal for
- People who want to run Llama 3 for chat or apps; this is an educational walkthrough, not a production inference server, and the README does not cover serving
- Readers without basic Python or linear algebra, since the explanations assume familiarity with tensors and matrix shapes
- Anyone looking for a complete implementation; the README says the BPE tokenizer is intentionally skipped and the truncated text covers only the first layer's early steps
How to get it
- 1
Clone and build from source
You'll need: Python 3 · Jupyter Notebook · PyTorch · tiktoken · matplotlib
Clone the repository
git clone https://github.com/naklecha/llama3-from-scratch - 2
Run it
- Download the Llama 3 weights from the official Meta link in the README and place them in the expected Meta-Llama-3-8B folder
- Open the notebook in Jupyter
- Run the cells in order to load the tokenizer and model file
- Check the printed token lists and tensor shapes against the explanations
- 3
Get the most out of it
- Read the diagrams in the images folder alongside each section to match shapes to operations
- Compare the tokenizer section with Andrej Karpathy's minbpe project, which the README links for a BPE implementation
- Print intermediate tensor shapes yourself to confirm each step's dimensions
Downloads
Strengths
- Clear, shape-by-shape explanations with diagrams that make the math easier to follow
- Uses the official Meta weights and config, so the numbers correspond to the real model
- Short, readable code that is easy to modify and experiment with
Trade-offs
- You must obtain the Meta weights separately, which requires accepting Meta's download terms
- The README is truncated and says the tokenizer is not implemented from scratch, so it is not a full standalone system
- Relies on torch and tiktoken, and the weights file is large, so it needs substantial local resources
How it compares
| Project | Stars | Type | Level | Updated | Best for |
|---|---|---|---|---|---|
| llama3-from-scratch (this) | 15.2k | Learning resource | intermediate | 2.5 years ago | Learners who understand basic Python and want to see how a transformer works internally |
| nanoGPT | 64.4k | Library | intermediate | 12 months ago | Developers who want to read a compact GPT training implementation |
| minbpe | 11k | Learning resource | intermediate | 2.3 years ago | Students and engineers who want to understand how LLM tokenizers work by reading code |
| llama2.c | 20.2k | Other | intermediate | 2.3 years ago | Developers who want to see how LLM inference works in plain C |
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.