Guide · Open source
How to use minbpe: train, encode and decode BPE tokens
Updated · written with AI from the project's own documentation and checked against our catalogue data
minbpe is a short, commented Python codebase that implements Byte Pair Encoding (BPE), the algorithm many large language models use to split text into tokens. This guide shows you how to clone it, train a tokenizer on your own text, convert text to token IDs and back, and where to look in the code. You do not need prior experience with tokenizers, but basic Python will help.
minbpekarpathy/minbpe · ★ 11k · MIT Install options & downloads Quick start
- 1Clone the repository. Run git clone https://github.com/karpathy/minbpe to get the source. You need Python and pip installed.
- 2Pick a tokenizer class. Use BasicTokenizer for plain BPE, or RegexTokenizer if you want category-based regex splitting and special tokens.
- 3Train on your text. Call train with your text and a vocab_size. Larger vocabularies need larger datasets.
- 4Encode and decode. Use encode to turn text into token IDs and decode to turn IDs back into text, then check the round trip.
- 5Save and load. Call save to write a .model file for loading and a .vocab file for viewing, then load the .model file later.
- 6Run the tests. Install pytest, then run pytest -v . from the repo to check the code and see more usage examples.
What minbpe is and what it does
minbpe provides three core tokenizer functions: training a vocabulary and merges on a given text, encoding text into tokens, and decoding tokens back into text.
The algorithm is byte-level, meaning it runs on UTF-8 encoded strings. The author cites the GPT-2 paper and code release as the popularizing source for LLMs, and says modern LLMs such as GPT, Llama and Mistral use this algorithm to train their tokenizers.
- Train: learn vocabulary and merges from a text
- Encode: turn text into token IDs
- Decode: turn token IDs back into text
Install minbpe by cloning the repository
The README's install method is to clone the repository. You need Python and pip. The README does not list a pip package or release for minbpe, so clone the source and work from it.
- Prerequisites: Python and pip
git clone https://github.com/karpathy/minbpeChoose the right tokenizer class
The repo contains three tokenizer classes built on a base class. BasicTokenizer is the simplest BPE implementation and runs directly on text. RegexTokenizer first splits text by a regex pattern into categories such as letters, numbers and punctuation, so no merges cross category boundaries, and it also handles special tokens. GPT4Tokenizer is a light wrapper around RegexTokenizer that the README says reproduces the tokenization of GPT-4 in tiktoken.
Use BasicTokenizer if you don't want regex preprocessing and don't need special tokens. Use RegexTokenizer if you want to follow how OpenAI splits text by category.
- minbpe/base.py: Tokenizer base class, meant to be inherited from
- minbpe/basic.py: BasicTokenizer
- minbpe/regex.py: RegexTokenizer
- minbpe/gpt4.py: GPT4Tokenizer
Run a small train, encode and decode example
The quick start reproduces the Wikipedia BPE example. minbpe always allocates the 256 individual bytes as tokens, then merges as needed. Training with 256 + 3 means 3 merges. The expected output for the sample string is the token list shown in the comment.
save() writes two files: a .model file for loading and a .vocab file for viewing.
from minbpe import BasicTokenizer
tokenizer = BasicTokenizer()
text = "aaabdaaabac"
tokenizer.train(text, 256 + 3) # 256 are the byte tokens, then do 3 merges
print(tokenizer.encode(text))
# [258, 100, 258, 97, 99]
print(tokenizer.decode([258, 100, 258, 97, 99]))
# aaabdaaabac
tokenizer.save("toy")
# writes two files: toy.model (for loading) and toy.vocab (for viewing)Train on your own text
For a custom tokenizer, train on your text with a chosen vocab_size. The README's example uses BasicTokenizer with vocab_size=4096 and shows save and load. To use regex splitting, use RegexTokenizer with the GPT-4 pattern as default. Change vocab_size to suit the size of your dataset.
Special tokens are added with register_special_tokens after training. The README says that with a vocab_size of 32768, the first special token must have id exactly 32768, which is the next id after the last merge token (32767). Encoding special tokens requires allowed_special, which can be "all", "none", or a list of tokens. The README warns that parsing special tokens from untrusted input, such as user prompts, can be a footgun, so declare your intent explicitly.
- BasicTokenizer: no regex splitting, no special tokens by default
- RegexTokenizer: regex splitting by category, supports special tokens
- allowed_special: "all", "none" or a list of special tokens
from minbpe import RegexTokenizer
tokenizer = RegexTokenizer()
tokenizer.train(very_long_training_string, vocab_size=32768)
tokenizer.register_special_tokens({"<|endoftext|>": 32768})
tokenizer.encode("<|endoftext|>hello world", allowed_special="all")Run the tests and the training script
The tests use pytest and live in the tests/ directory. Install pytest with pip, then run the tests from the repo root. The repo also includes train.py, which trains the two major tokenizers on tests/taylorswift.txt and saves the vocab for visualization. The README says this runs in about 25 seconds on the author's M1 MacBook. Your timing will vary.
$ pytest -v .Learn the algorithm with the exercise and lecture
The repo includes exercise.md, a step-by-step progression for building your own minbpe, and a lecture linked from the README, with a text version in lecture.md. The files are short and commented, and each one has a usage example at the bottom.
- exercise.md: step-by-step build exercise
- lecture.md: text form of the author's lecture
- Community: gnp/minbpe-rs, a Rust implementation with near one-to-one correspondence to the Python version
What people use it for
- Reading the code to understand how BPE merges are learned and applied
- Training a small custom tokenizer on your own text corpus
- Following the author's BPE lecture or the exercise.md progression
- Using RegexTokenizer as a hackable baseline before moving to heavier tooling
- Checking that a tokenizer matches the GPT-4 output in tiktoken for the test strings in the README
Frequently asked questions
Is minbpe free to use?
minbpe is released under the MIT licence, according to the repository listing. Check the LICENSE file in the repo for the full terms before you use it in a project.
How do I install minbpe?
The README's method is to clone the repository with git clone https://github.com/karpathy/minbpe. You need Python and pip. The README does not describe a pip package install.
What do I need to run minbpe?
The listed prerequisites are Python and pip. To run the GPT-4 comparison examples you also need to pip install tiktoken, and to run the tests you need pytest.
Can minbpe replace tiktoken for tokenizing text for an existing model?
For that job, tiktoken is the more direct choice. The README says minbpe is for learning and training your own tokenizer, and that the GPT4Tokenizer reproduces tiktoken's GPT-4 output, not that it replaces it.
Is minbpe fast enough for production or large datasets?
The README lists a more optimized Python version that could run over large files only as a todo, so minbpe is not positioned for production use on large corpora. Libraries such as Hugging Face tokenizers or sentencepiece are listed as comparable projects.
What is the difference between minbpe and sentencepiece or Hugging Face tokenizers?
The README lists tiktoken, sentencepiece and Hugging Face tokenizers as comparable projects but does not compare them in detail. minbpe's stated goal is minimal, readable code for learning BPE, so the choice depends on whether you want to read the algorithm or use a production tool.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use minbpe if you want to read a short, commented BPE implementation, train a small tokenizer on your own text, or follow the author's lecture and exercise. Pick tiktoken for tokenizing text with an existing model, or a more mature library for production or large corpora, since the README lists faster optimized code only as a todo.
Install options & downloads