Business cases for AI features are usually built on one number: the price per million tokens. It is the easiest number to find and the least important one in the model. In the engagements we have costed, inference is frequently under a third of the total cost of running the feature, and occasionally under a tenth.

Here is the structure we use instead, in the order the costs actually bite.

Inference, modeled properly

Token pricing is simple. Token volume is where estimates go wrong, usually by a factor of five or more.

The mistakes we see repeatedly:

  • Counting only the user's question. In a retrieval system, the retrieved context is typically ten to fifty times the size of the question. The system prompt, tool definitions and conversation history all bill as input on every single turn.
  • Forgetting that conversations resend everything. A ten turn conversation does not cost ten times a single turn. Without caching it costs closer to fifty times, because each turn resends the whole history.
  • Modeling the happy path only. Retries on malformed output, validation failures, and reasoning traces on models that produce them are all real tokens.

Two levers change this materially. Prompt caching cuts the cost of the stable prefix (system prompt, tool definitions, long shared context) substantially when the same prefix recurs, which in most applications it does on nearly every request. And routing by difficulty: sending the straightforward majority of requests to a smaller, cheaper model and escalating only the hard ones is often the single largest saving available, and it requires an evaluation suite to do safely, which is one more reason to build one.

Model, at minimum: input tokens per request including context, output tokens per request, requests per user per day, active users, cache hit rate, and a peak-to-average ratio for capacity.

Retrieval infrastructure

This is the cost line most often missing entirely from a business case.

  • Embedding generation. Cheap per token, but the initial index of a large corpus is a one-off charge worth knowing, and re-embedding after a chunking strategy change repeats it.
  • Vector storage and query. Managed vector databases price on stored vectors and queries per second. At small scale, pgvector on the Postgres you already run is often adequate and effectively free. At large scale a dedicated store earns its keep. Model both before committing, because migrating later is painful.
  • Re-ranking. A cross-encoder pass over thirty to fifty candidates per query is a real compute cost, sometimes comparable to generation itself. It is usually worth it, but it belongs in the model.
  • Ingestion. Document parsing, OCR for scanned material, and the pipeline that keeps the index fresh. OCR at volume is not cheap.

Evaluation

Building the golden set is a person-week of subject matter expert time, and that person is not cheap. Running the suite costs model calls on every change, plus the judge calls on top. Maintaining it as the product grows is ongoing.

Budget it explicitly. Teams that leave evaluation out of the cost model tend to conclude, once the invoices start arriving, that they cannot afford it, and then ship blind.

Human review

If your feature has an accuracy threshold it cannot yet meet unaided, someone reviews the output. This is frequently the largest line in the entire model and it is frequently omitted, because it is a people cost in a technology business case.

Model it honestly: what fraction of outputs need review, how many minutes each takes, and at what loaded hourly rate. Then model the trajectory, since the point of improving the system is that this fraction falls over time. A feature that needs 100% review at launch and 15% after six months has a very different total cost from one stuck at 100%, and the difference is worth designing for.

Engineering and operations

The build is a project cost and everyone plans for it. What gets missed is the year after:

  • Provider model deprecations, which arrive on the provider's schedule rather than yours and require a re-evaluation each time.
  • Prompt and retrieval maintenance as the underlying documents and business rules change.
  • Incident response, which for AI features means investigating a class of failure that is probabilistic and therefore harder to reproduce than a normal bug.
  • Observability infrastructure: tracing, logging and dashboards, all of which store more data per request than a conventional service.

A reasonable planning assumption is that the first year of operation costs a meaningful fraction of the original build. Treating an AI feature as finished at launch is the most expensive mistake on this list.

Governance

Depending on your sector and jurisdiction, some or all of: risk classification and documentation, bias and robustness testing, data protection impact assessment, security review, audit evidence collection, and periodic re-testing. If it is built into the delivery pipeline it is a modest ongoing cost. If it is retrofitted under deadline pressure because procurement asked a question nobody could answer, it is a project.

Putting it together

The output of this exercise should be a cost per unit of value, not a monthly total. Cost per resolved ticket, per document processed, per hour of analyst time returned. That is the number that can be compared against the thing it replaces, and it is the only form in which a finance function can evaluate it.

Two things fall out of doing this properly. The first is that some features are obviously worth building and you can say so with a number. The second, more often, is that a proposed feature is not viable at the volume intended, and you learn this during a costing exercise rather than after two quarters of engineering.

Both outcomes are worth the afternoon it takes.