This tool uses the cl100k_base tokenization scheme, which is the same encoding used by OpenAI's GPT-4, GPT-3.5-Turbo, and text-embedding-ada-002 models. It is implemented via the gpt-tokenizer JavaScript library, which produces token counts identical to OpenAI's tiktoken library.
The cl100k_base vocabulary is heavily optimized for Latin-script languages. Each common English word is often a single token, while Chinese, Japanese, Korean, and other non-Latin characters are typically split into two or three tokens per character. This is why the same semantic content expressed in Chinese generally costs 2–3 times more tokens than in English.
No. The entire tokenization process runs locally in your browser using the gpt-tokenizer JavaScript library. Your text is never sent to any server — not ours, not OpenAI's. This makes the tool safe to use with confidential prompts, private documents, or sensitive data.