Token
The bite-sized pieces of text an AI reads and writes: often a word, sometimes part of a word.
In everyday terms
"Unbelievable" might be split into "un", "believ", "able". Roughly, 1 token ≈ ¾ of an English word. Pricing and limits are counted in tokens.
For professionals
Sub-word units from a learned vocabulary (e.g. BPE). Each token maps to an ID and then an embedding vector.
Think of it like…
Lego bricks of language. The AI builds every sentence brick by brick.
You've already seen it
"Max 4,000 tokens", API prices per million tokens, odd spelling mistakes on rare words.
Myth vs reality
Myth: AI reads letters like we do.
Reality: It sees tokens. That's why it can struggle to count letters in a word.
Quick check
Roughly how many English words is 1,000 tokens?
- About 100
- About 750
- About 5,000
- Exactly 1,000
Show answer
About 750: 1 token ≈ 0.75 words.