Guide · Open source
How to use tiktoken: count and encode tokens in Python
Updated · written with AI from the project's own documentation and checked against our catalogue data
tiktoken is an open-source Python library that converts text into tokens, the numbers language models work with, using byte pair encoding (BPE). It can also turn tokens back into the original text. This guide shows you how to install it, encode and decode text, get the tokenizer for a specific OpenAI model, and count tokens before you send text to an OpenAI model.
tiktokenopenai/tiktoken · ★ 19.6k · MIT Install options & downloads Quick start
- 1Install the package. Run pip install tiktoken to install the library from PyPI. Python is the only prerequisite the README lists.
- 2Import tiktoken. Add import tiktoken at the top of your Python script or notebook.
- 3Choose an encoding. Use tiktoken.get_encoding with a name such as o200k_base, or tiktoken.encoding_for_model with a model name such as gpt-4o.
- 4Encode your text. Call enc.encode on your string to get a list of token numbers.
- 5Count or decode. Use len() on the encoded list to count tokens, or enc.decode to turn tokens back into text.
Install tiktoken with pip
tiktoken is published on PyPI, so you install it with the Python package manager. The only prerequisite named in the README is Python.
pip install tiktokenEncode and decode text with an encoding
Call tiktoken.get_encoding with an encoding name to get a tokenizer. The README uses o200k_base in its example. Encoding turns text into a list of token numbers, and decoding turns those numbers back into the original text. The README shows a round trip that returns the same string.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"Get the tokenizer for a specific OpenAI model
If you want the tokenizer that matches a particular model in the OpenAI API, use tiktoken.encoding_for_model with the model name. The README gives gpt-4o as an example. Use this when you need token counts that match the model you are calling.
enc = tiktoken.encoding_for_model("gpt-4o")Count tokens before sending text to a model
To count tokens, encode your text and take the length of the result. This lets you check how much input a prompt uses before you send it. The OpenAI Cookbook, linked from the README, has an example titled How to count tokens with tiktoken.
enc = tiktoken.encoding_for_model("gpt-4o")
num_tokens = len(enc.encode("Your text goes here"))Learn how BPE works with the educational submodule
The README describes BPE as reversible and lossless, able to handle arbitrary text, and usually producing about 4 bytes per token on average. For a hands-on look, tiktoken includes an educational submodule with code that helps visualise the BPE procedure. It can train a small BPE tokenizer and load the cl100k_base encoding for inspection.
- Train a simple tokenizer on a small amount of text with train_simple_encoding().
- Load the cl100k_base encoding into SimpleBytePairEncoding to see how text is encoded.
from tiktoken._educational import *
enc = train_simple_encoding()
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
enc.encode("hello world aaaaaaaaaaaa")Create custom encodings
The README describes two ways to extend tiktoken. The simpler one is to build a tiktoken.Encoding object yourself and pass it around in your code. The README warns that if you change the set of special tokens, you should use a different name so the behaviour is clear. The second way uses the tiktoken_ext plugin mechanism, which only helps if you need tiktoken.get_encoding to find your encoding. For that route, the README says to keep the tiktoken_ext/__init__.py file out of your project, and to avoid an editable install. The details are in tiktoken/registry.py.
- Option 1: construct a tiktoken.Encoding and pass it around. Prefer this unless you need get_encoding to find it.
- Option 2: register encodings through a namespace package under tiktoken_ext, with a ENCODING_CONSTRUCTORS dictionary in my_encodings.py, then run pip install ./my_tiktoken_extension.
What people use it for
- Counting tokens in a prompt before sending it to an OpenAI model, to check input size.
- Matching token counts to a specific OpenAI model such as gpt-4o by using encoding_for_model.
- Learning how byte pair encoding works using the educational submodule.
- Building a custom encoding with extra special tokens for your own OpenAI-style setup.
Frequently asked questions
Is tiktoken free to use?
tiktoken is released under the MIT licence, according to the project's listing. The README describes it as open source and installable from PyPI.
How do I install tiktoken?
Run pip install tiktoken. The README lists Python as the prerequisite and shows this as the install command from PyPI.
How do I count tokens with tiktoken?
Get an encoding with tiktoken.encoding_for_model, encode your text with enc.encode, and take the length of the returned list. The OpenAI Cookbook has a worked example linked from the README.
Can tiktoken be used for non-OpenAI models?
The README only describes tokenizers for OpenAI's models, so it does not claim to support other model families. If you need a tokenizer for a non-OpenAI model, look at tools such as Hugging Face tokenizers, SentencePiece or Transformers tokenizers.
How does tiktoken compare to Hugging Face tokenizers or SentencePiece?
The README says tiktoken is between 3-6x faster than a comparable open source tokeniser, measured on 1GB of text using the GPT-2 tokeniser against GPt2TokenizerFast from the Hugging Face tokenizers package. It does not compare tiktoken with SentencePiece, so check benchmarks for your own workload.
Where do I get help if tiktoken is not working?
The README asks people to post questions in the issue tracker on the openai/tiktoken GitHub repository. Include the encoding name and a short code sample that reproduces the problem.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
tiktoken suits Python developers who need to count or encode tokens for OpenAI models, especially when they want the tokenizer that matches a model such as gpt-4o. It is less suitable if you need a tokenizer for non-OpenAI models, work outside Python, or want a ready-made app with a user interface.
Install options & downloads