llama3 from scratch vs minbpe
Which should you use? A side-by-side look at what each one does, who it suits, what it costs, how hard it is to set up, and where each falls short.
Open source · MIT
A from-scratch walkthrough of Llama 3 inference in a Jupyter notebook, one tensor operation at a time.
Open source · MIT
A small, readable Python implementation of Byte Pair Encoding, the text-splitting method used in LLM tokenizer
The short answer
Choose llama3 from scratch if you need…
- Learners who understand basic Python and want to see how a transformer works internally
- Engineers who read model code and want a concrete, tensor-shape-level explanation
- Students studying attention and positional encoding
Choose minbpe if you need…
- Students and engineers who want to understand how LLM tokenizers work by reading code
- Developers who want to train a small custom tokenizer on their own text
- Anyone following the author's BPE lecture or exercise
- Researchers who want a hackable baseline before using heavier tooling
Side-by-side
| llama3 from scratch | minbpe | |
|---|---|---|
| Type | Learning resource | Learning resource |
| Price | Free and open source | Free and open source |
| Licence | MIT | MIT |
| GitHub stars | 15.2k | 11k |
| Setup difficulty | Intermediate | Intermediate |
| How you get it | Clone and build from source | Clone and build from source |
| Main language | Jupyter Notebook | Python |
| Last code update | 2 years ago | 2 years ago |
What each one is
llama3 from scratch
This is a Jupyter notebook that rebuilds parts of Meta's Llama 3 language model step by step, loading the official weight file and applying each operation (tokenizing, embedding, normalization, attention) by hand. It is meant to show how a transformer language model turns a text prompt into numbers and then computes with them. The README is truncated, so it covers only the early stages of the walkthrough.
minbpe
minbpe is a short Python codebase that implements Byte Pair Encoding (BPE), the algorithm that many large language models use to break text into tokens. It lets you train a tokenizer on your own text, encode text into token IDs, and decode those IDs back into text. The code is kept deliberately minimal and commented so people can read how the algorithm works.
Best for
llama3 from scratch
- Learners who understand basic Python and want to see how a transformer works internally
- Engineers who read model code and want a concrete, tensor-shape-level explanation
- Students studying attention and positional encoding
minbpe
- Students and engineers who want to understand how LLM tokenizers work by reading code
- Developers who want to train a small custom tokenizer on their own text
- Anyone following the author's BPE lecture or exercise
- Researchers who want a hackable baseline before using heavier tooling
Strengths
llama3 from scratch
- Clear, shape-by-shape explanations with diagrams that make the math easier to follow
- Uses the official Meta weights and config, so the numbers correspond to the real model
- Short, readable code that is easy to modify and experiment with
minbpe
- Very short, clean code with comments, designed to be read and modified
- Includes comparisons against tiktoken to verify behavior
- Covers the basic, regex-split, and GPT-4-compatible variants of the algorithm
Not ideal for
llama3 from scratch
- People who want to run Llama 3 for chat or apps; this is an educational walkthrough, not a production inference server, and the README does not cover serving
- Readers without basic Python or linear algebra, since the explanations assume familiarity with tensors and matrix shapes
- Anyone looking for a complete implementation; the README says the BPE tokenizer is intentionally skipped and the truncated text covers only the first layer's early steps
minbpe
- Production systems that need fast tokenization on large corpora, since the README lists a faster optimized version only as a future todo
- People who just need to tokenize text for an existing model, where a library such as tiktoken is the more direct choice
- Users who need a polished packaged tool with documented releases, since the repo has no listed releases
Trade-offs
llama3 from scratch
- You must obtain the Meta weights separately, which requires accepting Meta's download terms
- The README is truncated and says the tokenizer is not implemented from scratch, so it is not a full standalone system
- Relies on torch and tiktoken, and the weights file is large, so it needs substantial local resources
minbpe
- Not optimized for speed; the README lists a more optimized version as an unfinished todo
- Documentation is brief and assumes some familiarity with tokenization concepts
- Last push was 2024-07-01, so recent maintenance activity is unclear
Getting started
llama3 from scratch
Clone the repository
git clone https://github.com/naklecha/llama3-from-scratchminbpe
Clone the repository
git clone https://github.com/karpathy/minbpeFrequently asked questions
What is the difference between llama3 from scratch and minbpe?
llama3 from scratch: A from-scratch walkthrough of Llama 3 inference in a Jupyter notebook, one tensor operation at a time. minbpe: A small, readable Python implementation of Byte Pair Encoding, the text-splitting method used in LLM tokenizer. llama3 from scratch suits learners who understand basic python and want to see how a transformer works internally, while minbpe suits students and engineers who want to understand how llm tokenizers work by reading code.
Are llama3 from scratch and minbpe free?
llama3 from scratch: Free and open source. minbpe: Free and open source.
Is llama3 from scratch or minbpe more popular?
llama3 from scratch has more GitHub stars (15.2k) than minbpe (11k). Popularity isn't the same as fit — check which one matches your use case above.
What are other alternatives to llama3 from scratch and minbpe?
Other options people compare them with include llama2.c, tiktoken.
Keep comparing
Built from our catalogue entries for both products: open-source details come from each project's README (AI-assisted, refreshed weekly) and GitHub data; AI-tool details are editorial. Spotted something wrong? Let us know from either product's page.