Telling your team to stop using tokens may not be your smartest move

Leadership are increasingly interested in what useful productivity is being produced by the AI tokens billed with ROI questions. Meanwhile some users are burning tokens on hyper expensive models, still token maxing, whilst others are shy to consume any scared of the budget impacts as token prices rise, yet possibly having a negative impact on the company by not applying the intelligence they could be to problems. Yes burning tokens on flashy power point presentations for internal use may not be the best use, but not using AI where is should be is also an issue for companies trying to compete in the market place. Using AI (tokens) wisely to produce value is the new skill we all need to learn at all levels in an organisation.

The gigantic difference in AI consumption between someone writing emails with AI co pilots vs data analysis or vibe coding is important to acknowledge as most business user can be left to work guilt free with AI tools, the invoices are coming from the heavier users and shared systems built with AI. The dilemma is that these heavier use cases are often providing greatest value to the business too.

Back in the early days of AI, we didn't need to take much notice of tokens; they were someone else's concern. The AI companies were subsiding the cost. Token consumption was not on most people's radar, as back then your concern was the tiny context windows and how to get what we needed done in the space available. Yes, we were filling context with tokens, but it wasn't a consumption issue like it is today, as it was so constrained by context and the limited tooling.

That has changed with the rapid development of AI agents and huge, by comparison, context windows. Spawning multiple agents to burn a large number of credits is commonplace, and the focus has turned onto token use.

For the masses, AI moved from answering questions to deep research, agentic coding of software and running autonomous agents, and the token use that comes out of necessity is something businesses on both the consumer and service-provider sides of the relationship are now taking notice of.

Businesses were making mind-blowing targets encouraging employees to use substantial amounts of "AI", targets that many embraced with glee, consuming tokens from the firehose.

This has led to a new phase. When individuals in teams are consuming millions of tokens, the AI consumption invoices need to be paid and questions arise as to the efficient and effective use of that expenditure.

How do we get more useful work from the tokens we use?

Reducing consumption of tokens is not necessarily the challenge. Like most things in life that are a limited resource, which for the foreseeable future AI tokens are going to be anything but plentiful, we need instead to look at how to get the most out of what we use.

Let me back up. What is an AI token?

I've mentioned it before on this blog, but it's well worth exploring. The OpenAI language model tokenizer is well worth playing with. Enter text content and you will visually see how it is broken up into tokens, small fragments that represent meaningful, to AI, chunks of content. Use it to understand how a piece of text might be tokenised by a language model, and the total count of tokens in that piece of text.

One token very roughly corresponds to approximately four characters of text for common English text. I will talk about spoken languages in another post. This translates to roughly three-quarters of a word, therefore 100 tokens is approximately 75 words.

A token might be:

• an entire short word
• part of a longer word
• punctuation
• a number or part of a number
• whitespace or other fragments, depending on the tokenizer

Commercial AI is now metered by tokens, so these small units of measure become very important, especially for companies with large AI consumption needs, which I would argue should be most companies.

Note that you pay for AI to READ as well as WRITE

It is easy to assume that you only pay for tokens that are sent to you. Consumption is way more complicated than that. There are many different types of token that can all be billed differently and at different prices.

Suppose you ask AI to:

Summarise a contract in five bullet points...

You will receive, say, 150 words back. Doesn't feel like many tokens, right?

Feels low cost?

However, AI had to do a lot of processing to get that short reply:

• your 30k word contract
• your instructions
• previous messages in the conversation
• system instructions supplied by the application
• tool definitions
• retrieved documents
• additional context supplied automatically

In this example, you can easily see how the response is tiny but the "input" used to generate that response is enormous.

It is normal for input and output tokens to be priced independently for AI API use.

The size of the answer tells you remarkably little about how much work occurred to generate it.

Now for reasoning tokens...

With newer generations of AI, we are often provided with reasoning AI. This adds more complication. With reasoning models, they "talk to themselves" a lot before producing an answer. This ruminating can substantially add to the cost of usage.

While reasoning, the model is consuming reasoning tokens as it figures out the solution to the request, chattering away to itself. Reasoning is really effective at giving quality results, but at a token cost.

This is why comparing two AI platforms by comparing token cost is very naive. How those tokens are used is a crucial component of the pricing.

Image generation

This one surprised me... images are not necessarily the token beasts you might of expected. A "normal" image might consume hundreds or a few thousand input tokens, similar to a page or two of text, depending on the model and resolution. It is not the JPEG file size but how the AI provider converts the image into visual tokens or patches that influences token use. High res screenshots or repeated images in agentic workflows can cause costs to begin to compound.

One screenshot isn't especially worrying, but autonomous computer use agent taking, analysing and resending screenshots repeatedly can be.

Vibe coding & data analysis

Both coding and data analysis can be very expensive and token intensive, up and above x1000 that of writing emails using an LLM. It is hard to avoid this, due to the nature of the activity. You need lots of data held and passing through the AI context. Note that not all coding AI harnesses are created equal, there are reports of some being much more efficient (x2) at what they pass to the AI engine than others. Token efficiency is something worth investigating when selecting your developer tooling.

AI provider tokenizers

Each model provider have their own tokenisers.
The same document supplied to different providers could be 10-20% more tokens on one provider than another, due to differences in the way the tokenizer works.
I found this fascinating as I work in multiple European languages on web projects, but non -English text can have the potential to generate even larger differences.
Now this is significant as if you are comparing prices for tokens between providers

some providers require more tokens to do the same job as others so bear this in mind when comparing prices. There are examples of new models coming out with the same pricing but the tokenizer taking 30% more tokens to run same workload, although caching helped the costs a little. It is a little like shrinkflation, except instead of getting fewer biscuits in the packet, you're getting fewer words in your million tokens. This is one example of many where different feature sets interplay with each other making it hard to get a clear picture in a very rapidly changing product set.

I mentioned context earlier - the silent token consumer...

The LLM needs context, all the information surrounding the thing it is resolving. Indeed, it wasn't until larger contexts were developed that AI LLMs became much more capable. The model needs to hold a model of the conversation in context to know what you are talking about and to operate at all.

So previous conversation history is required, plus the foundation information to support the construction of the response, such as supporting "documentation".

I've talked about context windows in another post in detail, but a key understanding is that all your previous conversation history is resubmitted to the LLM with each further thing you "say" to it. So by the sixth back and forth, it will be carrying an ever-growing load behind it, building a substantial burden of context from what came before that latest prompt.

Good context hygiene is essential, starting new sessions with new tasks or when you recognise the context is getting too large. Recognise that context has a cost. A concise packet of relevant context can sometimes be both cheaper and better.

Although it is true that more context helps AI LLMs produce improved content, giving them everything is also not the best thing to do. Too much information stuffed into the same context makes the statistical relationships between the data more "blurred", less statistically strong or relevant, so there is a balance to be had.

This may be too much...

• Project documentation
• Meeting transcripts
• Database schemas
• Twenty-seven PDFs
• The company handbook
• Perhaps the takeaway menu from the office kitchen, just in case

It's both keeping the context precise and relevant, together with the fact that irrelevant content costs tokens too, that becomes the issue.

Give the model just enough context to do the job, not every piece of information you possess.

This is context engineering. It matters as much as prompt writing.

The agent revolution - what changed the economics completely...

Each of these stages involves tool calls and model calls. Each of those carries instructions, context, tool descriptions and results from earlier.

This is why we have seen how agentic AI is consuming dramatically more tokens (x5-x30) than ordinary chat. It is also why the AI companies had to quickly change how the pricing of AI worked when agents became popular.

A recent McKinsey analysis describes agentic workloads consuming vastly more tokens than simpler interactions and highlights refinement, checking, repairing and reverifying work, as being a surprisingly substantial part (60%) of the cost with agentic workloads.

Deep research and prompts that trigger reaching out to the web and other tooling can dramatically increase token usage. Some simple questions can consume insane amounts of tokens for bad answers with deep research vs simple model calls. The lesson is to be vigilant of what you are running.

Note that this is not really waste.

Going back to the premise that an AI agent spends $3 worth of compute investigating a software bug or technical fault that would otherwise occupy a £70-per-hour developer for half a day or more, no one can justify NOT spending the tokens, and this is where the wild consumption of AI is being driven.

I see this effect daily first hand. A model that look cheap can end up being a lot more expensive after it has taken many more interations and restarts than a more advanced, expensive model.

Choosing the correct model for the task is a key AI skill. Making the wrong choices can lead to overspend and maybe not even resolving the issue at all, instead circling in endless loops.

Companies have been benchmarking coding agents using real engineering tasks, discovering that evaluating price per completed task rather than price per token is the way to go.

• model choice
• agent design
• harness efficiency
• number of attempts needed

All can significantly affect the costs of completing the same work.

Conclusion: it is a much better metric.

How much does it cost us to get an acceptable result?

Include:

• model usage
• retries
• tool calls
• research
• corrections
• verification
• human review
• the employee's time

A model that charges more but reliably completes a task in one attempt may be significantly cheaper than a bargain model requiring repeated human intervention.

This is an important takeaway. Remember it.

Tokens are going to fade into the background with progress

We are going to see less about tokens due to AI products improving. A lot of these things are already happening;

1. Routers will choose models to use for a particular task.
2. Systems will cache prompts automatically for the user.
3. Applications will retrieve appropriate context, nothing more and nothing less as developers get better at managing it.
4. Agents will become better at recognising when they have finished their jobs rather than contemplating existence for another 300 million tokens, then starting again.

Running software is however going to be more like managing resource consumption rather than buying software seats. Someone or something needs to understand what it is consuming and why.

Example:
Give a traditional application ten customer records and it runs some code.

Give an AI system a difficult objective and it may decide to research, reason, retrieve documents, call tools, write code, inspect the result and ultimately try again -lol.

Don't optimise AI into uselessness, which is my fear of the danger of discovering token costs is becoming obsessed with reducing them.

The objective should be to maximum worthwhile outcome for the resources consumed. That might be by using fewer tokens, but perhaps it will mean a cheaper model, or the intelligent decision will be to use fewer tokens.

Maybe use a cheaper model where appropriate, alter the context so it stops guessing, but sometimes it will mean discovering that an AI agent has spent the weekend reading its own instructions 14k times and quietly switching the thing off.

You can see that comparing using tokens alone is not appropriate due to all the other factors involved, and outcome based costing is the only sensible approach.

Please employ a healthier strategy than just asking everybody to use slightly less intelligence (tokens) this month -because Accounts has seen the invoices.

In the next post I will talk about ways to reduce AI token waste, that will please the finance office...