What an AI feature costs to run

"What will it cost to run?" is usually answered with a per-token price, and a per-token price is not an answer. It is one input to an arithmetic problem that most teams do not do until the first invoice.

This is that arithmetic. Every number below is a variable you fill in — deliberately, because model prices change faster than any article about them stays true, and a figure quoted here would be wrong by the time you read it.

The unit is a completed task, not a token

The first mistake is measuring the wrong thing.

A user does not send one prompt. They send a request that becomes a retrieval call, two or three model turns, a tool call whose result goes back into the context, and a final answer. The cost you care about is per completed task, and it is usually several times the cost of the one model call people estimate.

The four lines on the bill

Two of those surprise people.

Retrieved context dominates input. The prompt you wrote is a few hundred tokens. The eight chunks you retrieved are several thousand. Input cost is a retrieval design decision far more than a prompt-writing one — and it is the reason reranking pays for itself: fewer, better chunks are cheaper and more accurate.

Output is priced above input, typically by a multiple. A verbose response format is a cost decision. So is asking for reasoning you then discard.

The worked example

Take one task. Fill in your own figures.

# Per completed task — your numbers, not ours.
input_tokens   = 4_200    # prompt + retrieved context + tool results
output_tokens  =   350    # the answer
turns          =     2    # model calls per task, including tool round-trips
retry_rate     =  0.05    # fraction of tasks needing a second attempt

price_in       = 0.000_000    # your provider's input price, per token
price_out      = 0.000_000    # your provider's output price, per token

per_turn  = input_tokens * price_in + output_tokens * price_out
per_task  = per_turn * turns * (1 + retry_rate)

tasks_per_month = 50_000
monthly = per_task * tasks_per_month

print(f"per task: {per_task:.4f}   monthly: {monthly:,.2f}")

Then run it three times: at today's volume, at ten times today's volume, and at the volume that would make the feature a success. The third number is the one that tells you whether the design survives, and it is the one nobody computes.

The four levers, in the order we pull them

1 · Retrieve less, better

The largest input cost is context you retrieved. Reranking down from twenty candidates to four cuts input tokens substantially and usually improves accuracy, because the model is no longer choosing between contradictory chunks. This is the rare optimisation with no trade-off.

2 · Cache the stable prefix

If your prompt has a large fixed preamble — a system prompt, a schema, a tool catalogue — provider prompt caching bills the repeated prefix at a reduced rate. It requires the prefix to be genuinely stable and to come first, which is a prompt-structure decision made once.

3 · Route by difficulty

Most tasks are easy. Sending all of them to your largest model is the single most common source of avoidable spend.

Route: a small model handles the clear cases, and escalates the rest. The engineering cost is a confidence signal you can trust and an evaluation set good enough to prove the routed system is not worse — which is real work, and worth it above a certain volume.

4 · Shorten the output

Constrain the response format. Structured output instead of prose, no restating the question, no summary of what it is about to say. Output tokens are the expensive ones.

What people forget

Embeddings are a re-indexing cost, not a one-off. Changing your chunking strategy or your embedding model means re-embedding the corpus. Budget for doing it more than once, because you will.

Observability scales with usage. Logging every prompt and completion is the right default and it is not free. Sample in production, keep failures in full.

Idle infrastructure still bills. A vector store, a queue and a couple of always-on services have a floor that does not care how many tasks you ran.

Evaluation costs money. Every suite run is inference. A good suite run on every change is a real line item — and much cheaper than the alternative.

The number to quote

When someone asks what an AI feature costs to run, the useful answer has three parts: cost per completed task, the volume it was computed at, and which lever you would pull first if that volume grew tenfold.

A price per million tokens is not a budget. It is one variable in the equation above.