mothertoken

installation

$ pip install mothertoken

rank tokenizers for a language

$ mothertoken rank ar

tokenizer id	chars/token	cost vs english
gpt-4o	3.132	1.56x ← most efficient
qwen2.5	2.585	1.84x
gpt-4	1.414	3.43x
mistral	1.104	3.85x

# aliases work too: arabic, spanish, thai, eng_Latn

list tokenizers

$ mothertoken list

id	name	used by	access	tokenizer
gpt-4o	OpenAI o200k_base	GPT-4o, GPT-4.1	local	tiktoken / o200k_base
qwen2.5	Qwen 2.5 tokenizer	Qwen 2.5	local	Hugging Face / Qwen/Qwen2.5-7B
claude-sonnet	Claude Sonnet tokenizer	Claude Sonnet	API	Anthropic / claude-sonnet-4-6
gemini-pro	Gemini Pro tokenizer	Gemini Pro	API	Google / gemini-1.5-pro

$ mothertoken list --local-only

# the benchmark measures tokenizers; model names are examples that help you pick the right id

tokenize exact text

 $ mothertoken tokenize "مرحبا بالعالم" --language ar --model gpt-4o 
   ModelTokensChars/TokenEnglish Est.Vs English 
  GPT-4o52.60031.56x 
  
 $ mothertoken tokenize --file prompt.txt 
 $ mothertoken tokenize "Hello" --include-api 
 $ mothertoken tokenize --file prompt.ar.txt --language ar --english-file prompt.en.txt 
 # local tokenizers run by default 
 # --include-api adds provider token counters when API keys are configured 
 # --language adds benchmark English-equivalent estimates 
 # --english-text or --english-file adds a paired translation comparison 

Model	Tokens	Chars/Token	English Est.	Vs English
GPT-4o	5	2.600	3	1.56x

researcher workflow

 $ mothertoken-benchmark --dry-run 
 $ mothertoken-benchmark --languages arb_Arab,spa_Latn --models gpt-4o,qwen2.5 
 # benchmark regeneration is documented separately from the everyday CLI 
 $  