What are tokens in AI prompts and how many is a word?

The short answer
A token in AI is a chunk of text, often part of a word, that a language model reads and writes. In English, one token is about 4 characters or three-quarters of a word, so 1,000 words is roughly 1,333 tokens. Tokens set what you pay and how much text fits in a ChatGPT or Claude chat.
What is a token in AI?
A token in AI is a chunk of text, often part of a word, that a language model reads and writes as one unit. It can be a whole word, part of a word, a number or a punctuation mark. A model doesn't read letters or words directly: a tokenizer first splits your text into tokens, and the model works with those pieces.
Exact splits depend on the model's tokenizer, so don't expect the same split in every model. A common pattern splits the word "tokenization" into 2 tokens: "token" and "ization". Short, common words are usually 1 token, and rare or long words split into 2 or more. Every token is either an input token (your prompt, files and history) or an output token (the model's reply), and the two are counted separately.
How many AI tokens is a word, a page or a book?
One English word is about 1.33 AI tokens, so 1,000 words is roughly 1,333 tokens and a 300-word page is roughly 400 tokens. OpenAI's help center gives the rule of thumb behind it: one token is about 4 characters, or three-quarters of a word, and 100 tokens is about 75 words. The table applies that rule and assumes pages of 300 words, so its figures are rough.
| Text | Words | Estimated tokens |
|---|---|---|
| A short email | 150 | about 200 |
| A blog post | 1,000 | about 1,333 |
| A 100-page report | 30,000 | about 40,000 |
| A novel | 80,000 | about 107,000 |
Where the tokens per word rule breaks
The tokens per word rule breaks in 3 places: code and symbols, languages other than English and newer tokenizers that split text into more pieces.
- Code and symbols: brackets, indentation and long numbers split into more tokens per character than prose does.
- Other languages: most languages need more tokens than English for the same meaning.
- Newer tokenizers: Anthropic's documentation says Claude 4.7 and later models use a newer tokenizer that produces about 30 percent more tokens for the same text than earlier models.
There's no fixed ratio that holds across models, so treat 1.33 tokens a word as an English estimate only.
Why do tokens matter for cost, context and output length?
Tokens matter for 3 reasons: Application Programming Interface (API) cost per input and output token, the size of the context window and the cap on answer length.
- Cost: API pricing lists an input token price and an output token price separately, so a long prompt and a long answer are both billed.
- Context: the window caps how much the model can hold at once. Past it, an app drops or summarises older turns, and an API request can be rejected.
- Output length: every model has its own cap on one reply, so a request for a book-length document may come back cut short.
Chat plans such as Claude's meter usage too. Limits reset every few hours or every week, and heavy prompts use your allowance faster. Prices change often, so this page keeps the unit and leaves the dollar figures to each vendor's own page.
What is a context window in tokens?
A context window is the most tokens a model can hold in one request, counting your prompt, files, chat history, system prompt and its own answer. It's measured in tokens, not words, and its size differs by model.
Everything in the conversation counts toward it: the messages you've typed, the replies you've had, every attached file, the standing system text and the answer that's being written now. That's why a long chat with big files can hit the limit even when your newest message is short.
How do you count tokens before sending a prompt?
To count tokens before sending a prompt, paste the text into the vendor's tokenizer or read the token count the API returns with each response.
- Browser tokenizer: OpenAI's Tokenizer tool and the tiktoken library show exact counts for OpenAI models.
- Model counter: Anthropic's token counting endpoint takes the same input as a message, including system prompts and tools, and returns the input tokens. It's free to use, with rate limits, and the count can differ by a small amount from Claude billing.
- API usage fields: every response reports the input and output tokens it used, so you can log the real numbers.
- Quick estimate: divide the characters by 4, or multiply the words by 1.33.
The token ruler on the home page shows an estimate for any prompt you've built, measured by the site's own engine. An estimate isn't an exact count, so recount whenever you switch models, say from GPT to Claude.
How many tokens do a system prompt and custom instructions use?
A system prompt and custom instructions use the same tokens as any text, and they're counted again on every message in the conversation. They sit at the start of the context, and the model reads them with each request, so a 500-token system prompt in a 20-message chat is read 20 times.
That's the cost most token explainers skip: a long standing instruction costs on every turn, not once. Anthropic's counting endpoint includes system prompts and tool definitions in its total, which makes the extra visible. Keep standing text short, and put one-off details in the message where they're needed once. The system prompt generator builds compact standing text, and the custom instructions generator saves it for chat apps.
What are tokens in AI before and after
The same request, asked two ways. Read the second one slowly. Each line does a job.
Hi! I hope you're doing well. I was wondering if you could possibly help me write a summary. Please make sure the summary is good and clear and easy to read, and please keep it short, and make sure that it is clear. The text is below, please read it carefully and summarise it for me.
Summarise the text below in 3 bullet points of 20 words or fewer each. Use plain words. Keep names and numbers exactly as written.
Why it works
The rewrite states the length, the format and one constraint once, and drops the greeting and the repeated "clear", so it uses about a quarter of the tokens and gives the model more to act on.
What are tokens in AI examples
Shorten a long prompt without losing facts
The removed-items list lets you restore anything that mattered.
Rewrite the prompt below so it is about {{target}} percent of its current length. Keep every fact, number, name, constraint and output format. Remove repeats, greetings and filler. Show the shortened prompt first, then list anything you removed that might change the answer. Prompt: {{prompt}}
How to use
- Copy it with the button above.
- Fill the blanks in {{double braces}} with your own details.
- Paste it into ChatGPT, Claude or Gemini and send it.
- Then follow up: ask for one change at a time.
Before you run “Shorten a long prompt without losing facts” for the first time, the lesson on how to use AI prompts step by step walks through copying, filling and sending it.
Lay out a long task with the data at the end
Putting the data in one labelled block lets you swap it without rewriting the instructions.
Task: {{task}} Output format: {{format}} Rules: {{rules}} Use only the data in the block labelled DATA below. If the data doesn't contain the answer, say so. DATA: {{data}}
How do you cut prompt tokens without losing output quality?
To cut prompt tokens without losing output quality, remove repeated background, keep one short example and state the output format once. You can do the rewrite in one step with the prompt improver, the site's prompt optimizer, which makes a shorter prompt and keeps its structure, so the same output quality is easier to hold.
- Delete repeated instructions. Say each rule once, where it's needed.
- Drop pleasantries and any background the model can't use.
- Keep every fact, number, name and constraint, since they're what change the answer.
- Attach long data as a file, or paste it in one block at the end, and refer to it by label.
- Replace long examples with one short example that shows the format.
A prompt can run from 20 tokens for a one-line question to 2,000 tokens for a task with data, rules and examples. Test each sentence: if you deleted it, would the answer get worse? A shorter prompt isn't automatically better, so check the answer after each cut and don't trim facts. The context engineering guide covers what belongs in the window and what doesn't.
What are tokens in AI FAQ
Still stuck? Build it in Prompt Studio. It'll score each part as you go.
How many tokens is 1,000 words of English text?
1,000 words of English text is about 1,333 tokens, using the rule of three-quarters of a word per token. Code, numbers and other languages need more, and GPT and Claude tokenizers differ, so count with the model's own tool.
Do spaces and punctuation count as AI tokens?
Yes, spaces and punctuation count as AI tokens, and the space before a word usually joins the token of the word after it. Punctuation marks are often their own tokens, and text with deep indentation, such as Python code, can use extra ones.
Why do models count different numbers of tokens for one text?
Models count different numbers of tokens for one text since every model family ships its own tokenizer with its own split rules. Each one has its own vocabulary of chunks, so last month's count won't carry over to a new model.
Are AI tokens the same as characters?
No, AI tokens aren't characters: one English token averages about 4 characters, and short common words are a single token. A token can be one character or a whole word, which is why characters divided by 4 is only an estimate.
Does a longer prompt always cost more tokens?
Yes, a longer prompt always costs more tokens, as input tokens, and a longer answer costs more output tokens on top. Standing text such as a system prompt adds its tokens to every message, so trim it first.