Tokens and context windows: why AI forgets and what it costs

You paste a 40-page contract into an AI assistant and ask about the termination clause. The answer is confident and quotes the wrong section. Or a chat that went well for an hour starts ignoring the rules you set at the beginning. Or the first monthly bill for a new AI feature is three times what someone estimated.

All three come back to two ideas: tokens and the context window. You don’t need to be technical to understand them. Once you do, you’ll make better decisions about how your team uses chat assistants, what to build, and what AI will cost to run.

What a token is

Language models don’t read letters or whole words. They read tokens: chunks of text from a fixed vocabulary the model learned during training. (If you’re new to how models work, start with what generative AI is.)

• Common English words are usually one token: invoice, Friday, payment.
• Longer or rarer words are split into pieces: overpayment becomes over and payment.
• Numbers are cut into chunks of a few digits, which is one reason models are clumsy with arithmetic.
• Spaces and punctuation count too.

OpenAI’s rule of thumb for English: one token is about four characters, or about three-quarters of a word. So 100 tokens are roughly 75 words, and a full page of about 500 words is roughly 650 tokens.

How a model sees a sentence

Text

Please reimburse the overpayment on invoice 2026-0147 by Friday.

Tokens

Please reimburse the over payment on invoice 202 6 - 014 7 by Friday .

Why Slovak, Czech and German cost more

Tokenizers are built from training text that is mostly English, so English gets the most efficient pieces. Other languages split into more, shorter tokens. Slovak, with its endings and diacritics, splits a lot: where overpayment is two tokens, the Slovak preplatok is three, and vrátime (“we will refund”) is three more.

The same email in four languages

• English 65 tokens
• German 79 tokens
• Czech 111 tokens
• Slovak 122 tokens

In practice, the same work in Slovak can use about twice the tokens of English. That means roughly twice the cost per request, and a context window that fills up about twice as fast. It’s not a reason to avoid Slovak. It is a reason to estimate with your real texts, in your real languages, rather than with an English rule of thumb.

How to count tokens yourself

You don’t have to guess. OpenAI has a free tokenizer page on its website, and its tiktoken library is open source. Anthropic and Google let developers count tokens through their APIs before sending a request. Every major provider also reports the tokens used by each request, so your developers can log them from the first day of a pilot. Use those logs, not a rule of thumb, for any estimate that ends up in a budget.

How tokens decide the price

AI providers bill by the token, usually quoted per million tokens. A few things about that bill surprise people:

• Input and output are priced separately. Input is everything you send. Output is what the model writes. Output tokens usually cost several times more than input tokens.
• Every request pays for everything in it. The instructions, the documents, the conversation history and the tool results are billed again with every new message, not once.
• Thinking costs money. Models that reason before they answer produce extra tokens you don’t see but do pay for, as output.
• Agents multiply it. An AI agent works in a loop, and each step resends the growing context. One agent run can use many times the tokens of a single call. That’s one of several reasons to check whether you need an agent at all.

Here is how to estimate it without knowing today’s exact prices. Take a support team whose assistant drafts replies from the help centre:

A worked estimate

• 4,500 input tokens per request: instructions, help-centre passages, the customer’s email
• 300 output tokens per request: the drafted reply
• 3,000 replies drafted a month
• 14.4M tokens a month: 13.5M in, 0.9M out

Depending on the model you choose, an example like this lands anywhere from a few euros to a few hundred euros a month. For most business processes, tokens are the smaller cost. Building the integration and the time people spend reviewing outputs usually cost more. We break that down in what an AI project really costs.

What the context window is

The context window is everything the model can see in one request, measured in tokens. It’s not just your question. The instructions, any documents, the conversation so far, the results of tool calls and the answer the model is writing all have to fit.

What fills the context window

One request ↓ · Answer ↑

1. Instructions Role, rules, format: often called the system prompt
2. Saved memory Notes your app kept from earlier conversations
3. Documents and search results Contract pages, help-centre passages
4. Tool results Order status, a CRM record, database rows
5. The conversation so far Every earlier question and answer
6. The latest question
7. Room for the answer The output counts towards the limit too

How big is it? Current models from OpenAI, Anthropic and Google accept hundreds of thousands of tokens, and some around a million. Open-weight models that you run on your own servers often work with less, and a longer context needs more memory on your hardware. That matters if you’re weighing self-hosted models against API models.

To get a feel for the sizes (English text, rounded):

Material	Size	Rough tokens
A short email	150 words	200
A one-hour meeting transcript	9,000 words	12,000
A 20-page contract	10,000 words	13,000
A 100-page operations manual	40,000 words	53,000
A year of a small support team’s emails	5,000 emails	1,000,000

The table uses OpenAI’s three-quarters-of-a-word rule; the same material in Slovak would need roughly twice as many tokens. A contract or a manual fits easily. Your whole email archive does not, and it shouldn’t have to.

Bigger is not automatically better

A large window is useful, but filling it has a price:

• Money. Every token in the window is billed on every request.
• Speed. More input means a slower answer.
• Attention. Models use information at the start and the end of a long input better than information buried in the middle. Researchers from Stanford and elsewhere showed this in a 2023 paper called “Lost in the Middle”. The more irrelevant material you add, the easier it is for the model to miss the one paragraph that matters.

Back to the contract at the top of this article. All 40 pages fit in the window, so size wasn’t the problem. The termination clause was one paragraph among hundreds, the question was vague, and nothing asked the model to show where its answer came from. Ask it to quote the clause word for word, with the section number, and a wrong answer becomes easy to spot. Better still, have your system find the right section first and send only that.

Why AI “forgets”

Within one conversation: long chats drift

A chat feels like an ongoing conversation, but the model keeps no state between messages. Each time you send one, the app sends the whole conversation again. When it gets too long for the window, the app drops or condenses the oldest parts. Even before that, your instructions from the start get diluted by everything that followed.

A recruiter starts a chat with “always write in formal Slovak and never mention salary”. Forty candidate emails later, a draft mentions salary. The model didn’t disobey on purpose. That instruction is now a small, distant part of a very long input.

Between conversations: it doesn’t remember yesterday

The model itself has no memory. Each request starts from zero, and the model knows only what is in its context window at that moment. When a chat assistant “remembers” your preferences, the app has saved notes and quietly adds them to the context. In your own systems, memory means storing what matters (the customer’s history, past decisions, open issues) in a database and adding the relevant part to each request.

Practical tips

For people using chat assistants:

1. Start a new chat for each task. Carry over a short summary, not the whole thread.
2. Put the rules at the start, and repeat the critical ones. With long documents, paste the document first and ask your question at the end.
3. Give only what’s needed. For a narrow question, the three relevant pages of a contract beat the whole contract.
4. Ask for short answers. “Five bullet points” costs less and is easier to check than “explain everything”.

For teams building AI into their tools:

1. Retrieve, don’t paste. Search your documents and send only the relevant passages, not the whole manual. That’s the idea behind RAG.
2. Trim tool results. If the model needs five fields from a customer record, don’t send all 300.
3. Keep a running summary of long cases instead of the full history.
4. Keep fixed instructions identical and at the start. Providers offer prompt caching, a discount when the beginning of a request repeats exactly.
5. Cap the loops. Limit how many steps an agent may take and how much it may spend on one task.
6. Measure with real examples. Run 20–50 real requests and log the token counts. You’ll get a better estimate than any rule of thumb.

What to do next

• Pick one AI task your team runs often and count its tokens per request: instructions, documents, history, answer.
• Check which languages your texts are in and adjust the estimate.
• Look at what goes into each request and remove what the model doesn’t need.
• Multiply by monthly volume and the current price list before you commit to a model.

If you’re working out what an AI feature would cost to run on your own documents and volumes, we’re happy to help you test it. Get in touch.