[ back to lab ]
Tokenizer comparator
Same text, three tokenizers, three different breaks. The first place a model can fail to understand you — and the place where prompts get expensive without you noticing. Runs entirely in your browser.
GPT-4o
o200k_base[ loading ]
[ loading ]
GPT-4 / 3.5
cl100k_base[ loading ]
[ loading ]
Llama 3
tiktoken-style BPE[ loading ]
[ loading ]
Claude and Gemini don’t publish their tokenizers, so they can’t run in a browser. Everything here runs on this page: no network calls, no API keys, no tracking.