The Fundamental Unit of Large Language Models
When you use ChatGPT, Claude, or Cursor, you frequently encounter the word "token":
- "Context window: 200,000 tokens"
- "Pricing: $3.00 per million tokens"
- "Max output: 8,192 tokens"
Tokens are the fundamental currency of Large Language Models. Understanding what tokens are and how they are counted is essential for estimating API costs, optimizing prompt performance, and avoiding usage limits.
This guide provides an accessible explanation of AI tokens in 2026. For a deeper technical breakdown of token algorithms, see how AI tokens work and prompt tokens vs output tokens.
What Exactly Is a Token?
Human beings read text by recognizing words. AI models, by contrast, read text by recognizing tokens—numerical chunks of characters.
A token can represent:
- A whole common word:
"apple"(1 token) - Part of a longer word:
"un"+"forget"+"table"(3 tokens) - A punctuation mark:
"?"or"!"(1 token) - A series of spaces or code indentation:
" "(1 token)
The Golden Rule of Thumb
In standard English prose:
- 1 Token ≈ 4 characters (including spaces)
- 1 Token ≈ 0.75 words
- 100 Tokens ≈ 75 words
- 1,000 Tokens ≈ 750 words
Why AI Uses Tokens Instead of Words
Why don't AI creators just train models on whole words?
If a model only understood whole words, its dictionary would need millions of entries to cover every technical jargon term, surname, and typo. If it encountered an unseen word ("PromptCrafting"), it would fail completely.
By utilizing sub-word tokenization, models can assemble any arbitrary string from a manageable vocabulary of approximately 50,000 to 128,000 standard token building blocks.
Token Consumption: Prose vs Code
A critical insight for software engineers is that programming code is far more token-dense than standard English text.
Consider this comparison:
| Text Type | Sample Text | Word Count | Approximate Tokens | Why the Difference? |
|---|---|---|---|---|
| English Prose | "The quick brown fox jumps over the lazy dog." | 9 words | 9 tokens | Common words map 1:1 to single tokens |
| JavaScript Code | const [user, setUser] = useState(null); | 7 words | 18 tokens | Brackets, commas, camelCase, and punctuation become separate tokens |
| JSON Payload | {"status": 200, "success": true, "items": []} | 5 words | 14 tokens | Quotation marks, colons, brackets, and braces each consume tokens |
Because code syntax contains numerous symbols and custom identifier names, uploading 1,000 lines of code consumes far more context than 1,000 lines of a novel.
How Tokens Determine API Pricing
AI cloud providers bill for computation based on token volume:
- Input Tokens: The prompt you provide (including any uploaded documents, system prompts, and previous chat history).
- Output Tokens: The response the AI model generates.
Output tokens typically cost 3x to 5x more than input tokens because generating tokens sequentially requires significantly more GPU compute cycles. Learn more in our breakdown of Claude token limits.
Looking to build an AI-powered web app with predictable API costs? Explore our full-stack web development services or view our pricing plans.
Frequently asked questions
What is a token in simple terms?
A token is a fragment of a word that an AI model reads and writes. In English, one token is approximately 4 characters or roughly 0.75 words.
Why do AI models use tokens instead of full words?
Using full words would require a dictionary of millions of entries that could not handle misspellings, new slang, or code syntax. Tokens break words into modular sub-words, allowing models to represent any text with a vocabulary of roughly 100,000 tokens.
How do tokens affect AI pricing?
All major AI APIs (OpenAI, Anthropic, Google) charge per million tokens processed. Input tokens (what you send) are billed at lower rates than output tokens (what the model generates).
Do spaces and punctuation count as tokens?
Yes. Punctuation marks, special symbols, and leading spaces often count as individual tokens. In programming code, indentation spaces and brackets consume significant tokens.