Which OpenAI models use o200k_base and which use cl100k_base
OpenAI's tokenizer library, tiktoken, maps model names to encodings in one file, tiktoken/model.py. As of the 2026-08-17 revision of that file, the mapping for current models is:
| Encoding | Models (exact names or name prefixes in model.py) |
|---|---|
| o200k_base | Any name starting gpt-5; gpt-4.5-; gpt-4.1; gpt-4o and chatgpt-4o-; o1, o3, o4-mini; fine-tuned ft:gpt-4o |
| cl100k_base | gpt-4 and gpt-4- variants (including gpt-4-turbo); gpt-3.5-turbo; text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002; davinci-002, babbage-002 |
| o200k_harmony | gpt-oss- open-weight models. It uses o200k_base's vocabulary and splitting rules and only adds chat-formatting special tokens, so the o200k_base count above applies to plain text. |
If a model you use is not in that file, tiktoken refuses to guess and raises an error. Check the model's documentation before assuming an encoding.
What a token looks like, with real examples
A token is an entry in the model's vocabulary: often a whole common word with its leading space, sometimes a word fragment, a punctuation mark or a single byte. Every count below was produced by this page's tokenizer, and you can reproduce any of them by pasting the text above and turning on the token view.
| Text | Characters | o200k_base | cl100k_base |
|---|---|---|---|
hello world | 11 | 2 | 2 |
The quick brown fox jumps over the lazy dog. | 44 | 10 | 10 |
2026-10-26 | 10 | 6 | 6 |
{"id": 42, "name": "Ada"} | 25 | 12 | 12 |
Größenverhältnis | 16 | 5 | 8 |
Привет, как дела? | 17 | 6 | 8 |
日本語のテキスト | 8 | 6 | 8 |
| 👍 | 1 | 1 | 3 |
Two things stand out. For plain English the encodings tie, so moving an English prompt from GPT-4 to GPT-4o barely changes its count. For German, Russian, Japanese and emoji, o200k_base is clearly cheaper, because its larger vocabulary gives more of those character sequences their own entry. Dates and JSON punctuation cost more than their length suggests: 2026-10-26 is 6 tokens for 10 characters. The guide JSON vs CSV vs YAML: token cost measured applies this to whole datasets.
Counting tokens for Claude and Gemini
Anthropic and Google do not publish the tokenizers behind their current models, so no client-side tool can give you their exact numbers. What they do publish is a rough ratio and a counting API:
- Gemini: Google's token documentation says a token “is equivalent to about 4 characters” and that 100 tokens are “about 60-80 English words”. Exact counts come from the
countTokensmethod of the Gemini API. - Claude: Anthropic's glossary says a token represents about 3.5 English characters on models before Claude Opus 4.7, and that Opus 4.7 and later use a newer tokenizer that produces about 30% more tokens for the same text. Exact counts come from the
/v1/messages/count_tokensendpoint, which Anthropic lists as free to use, with its own rate limits.
The estimate panel in the tool applies those ratios literally to your character count. Treat them as a sanity check for English prose. For a budget or a context-window limit, call the vendor's endpoint with the exact model ID.
What the count does not include
This page counts the text you give it, nothing more. A real API request costs more than its text:
- Message framing. Chat requests wrap every message in a few formatting tokens. OpenAI's cookbook example adds 3 tokens per message plus 3 to prime the reply for the models it covers, and says outright that the method may change between models and is an estimate.
- Tools and images. Function or tool definitions are added to the prompt and billed as input tokens. Images are billed under their own rules, which this page does not model.
- Output. Generated tokens are billed separately, often at a different rate from input. The cost box multiplies whatever price you enter by the count you pick, so enter the input rate for a prompt.
How to check these counts yourself
OpenAI's reference implementation is the Python package tiktoken. With Python installed, this prints the token IDs for hello world in cl100k_base:
pip install tiktoken
python -c "import tiktoken; print(tiktoken.get_encoding('cl100k_base').encode('hello world'))"Expected output: [15339, 1917], which is also an assertion in tiktoken's own test suite. Turn on the token view above, pick cl100k_base and hover over each highlighted piece of hello world: the tooltips show the same two IDs. For o200k_base the IDs are [24912, 2375].
We tested this page against tiktoken's published cl100k_base test vectors. All matched except one: text containing the control character U+0085 (NEXT LINE) can count one token differently. JavaScript's regular expressions don't treat U+0085 as whitespace and Rust's do, and every JavaScript tokenizer port we checked behaves the same way. Ordinary text never contains this character.
Frequently asked questions
- Is my text sent to a server?
- No. The tokenizer runs in a Web Worker inside your browser tab. The page downloads two encoding files from larely.com the first time you count something, and after that it makes no network requests at all, so you can disconnect and keep counting. Your text is never transmitted or stored.
- Why do GPT-4o and GPT-4 give different counts for the same text?
- They use different encodings. GPT-4o, GPT-4.1, GPT-5 and the o-series use o200k_base, a vocabulary of about 200,000 tokens. GPT-4 and GPT-3.5 Turbo use cl100k_base, about 100,000 tokens. English prose usually counts about the same in both, but o200k_base needs noticeably fewer tokens for many other languages and for emoji, because more of those pieces have their own vocabulary entry.
- Can this count tokens for Claude or Gemini?
- Not exactly. Anthropic and Google do not publish the tokenizers their current models use, so no browser tool can reproduce their counts. The page shows each vendor's own published rule of thumb, clearly labelled as an estimate. For an exact number, call Anthropic's free count_tokens endpoint or Google's countTokens method with the model you plan to use.
- Does the count include the system prompt and chat formatting?
- It counts exactly the text you paste. A chat request adds a few formatting tokens around every message, and tool definitions and images cost extra. OpenAI's own token-counting example treats those additions as model-specific estimates, so paste your system prompt and messages together and allow a small margin.
- Why do emoji and non-English text use more tokens?
- These encodings work on UTF-8 bytes. Common English words are single vocabulary entries, while a character outside the most frequent sequences may take two or three byte-level tokens. In cl100k_base the thumbs-up emoji is 3 tokens; in o200k_base it is 1. Turn on the token view above to see exactly where your text is split.
Sources
- OpenAI tiktoken,
tiktoken/model.py(revision of 2026-08-17): the model-to-encoding mapping in the table above - OpenAI tiktoken,
tiktoken_ext/openai_public.py: shows o200k_harmony reusing o200k_base's ranks and pattern - OpenAI tiktoken,
tests/test_encoding.py: the cl100k_base test vectors this page was checked against - OpenAI Cookbook: How to count tokens with tiktoken: per-message formatting overhead and its caveats
- Anthropic: Token counting and Anthropic glossary, “Tokens”: the count_tokens endpoint, the 3.5-character figure and the newer tokenizer
- Google AI for Developers: Understand and count tokens: the 4-character figure and countTokens
- gpt-tokenizer (MIT): the JavaScript tokenizer this page runs, version 3.4.0, served from larely.com