Tokens: What Your Agent Actually Reads

Type a sentence to an AI and, before anything else happens, the sentence is chopped up. Not into letters. Not into words. Into tokens — chunks of characters that the model learned to treat as units, the way you read “смартфон” as one thing rather than nine letters.

A token is usually a short common word, a piece of a longer word, a space glued to the word after it, or a punctuation mark. In English, one token is on average about ¾ of a word. The word cat is one token; bookkeeping might be three; a long chemical name might be seven.

Why you should care at all: everything about your agent is measured in tokens — how much it can hold in mind at once, how fast it answers, and how quickly your subscription allowance runs down. Tokens are the currency. You don't need the math; you need a feel for the exchange rate.

Get a feel for it

Type anything below — or press the presets. Each colored chunk is roughly one token.

Token feel — approximate

characters

words

≈ tokens

—

tokens per word

This is an approximation for intuition — real tokenizers learned their chunks from data and differ per model. For the exact version, try tiktokenizer.vercel.app.

Try the Russian preset and watch the count. The same sentence costs noticeably more tokens in Russian than in English — most models learned mostly English text, so they chop Cyrillic into smaller pieces. Nothing is wrong with your writing; the exchange rate is simply worse. It's one reason long Russian documents fill an agent's memory faster.

The three places tokens touch your day

1. The desk. An agent's working memory — its context window — is measured in tokens: everything you say, every file it reads, everything it answers, piles onto one desk of fixed size. A 100-page document can eat the whole desk by itself. This is why, later in the course, you'll learn to point the agent at the three relevant files instead of the whole folder.

2. The meter. Your subscription allows a certain amount of agent work per 5-hour window. Long pasted texts and “try again, again, again” loops spend it fastest. One well-described request is cheaper than five vague ones — in tokens, in minutes, and in patience.

3. The speed. The model produces its answer token by token, like a fast typist. Longer answers take longer. When you ask for “a short summary, 5 bullets” you're not just saving your reading time — you're saving generation time too.

Rule of thumb English: 1 token ≈ ¾ of a word, so 1,000 tokens ≈ 750 words ≈ 1.5 pages. Russian: closer to 1,000 tokens ≈ 400–500 words. A number, a code, or an unusual name costs more than it looks.

Check yourself

Your colleague pastes a 60-page contract into the agent “so it has all the context.” What just happened, in token terms?

The desk filled up. Tens of thousands of tokens now sit in the agent's working memory, most of them irrelevant to the actual question — leaving less room, less attention, and a slower, more expensive session. Better: ask the agent to read just the relevant section, or ask for a summary first.

True or false: a token is the same thing as a word.

False. A token is a learned chunk of characters: short common words are one token, long or rare words are several, and Russian words are usually more pieces than English ones. That's why token counts and word counts never match.

You never need to count tokens by hand — the agent does that. What you need is the instinct: text has weight. Big pastes are heavy, focused requests are light, and the person who feels that difference gets more out of the same subscription.