Choosing an AI model can look like a product-comparison exercise. One provider leads a benchmark. Another cuts its price. A third adds a larger context window or a faster reasoning mode. Within weeks, the apparent order changes.

The lasting decision is not which model is best. It is how the business will decide what is good enough for a particular task, what the result costs in practice and when a different option should be used.

That moves procurement away from model loyalty and towards operating discipline.

Token price is an input, not the answer

Two models with the same published price can consume different amounts of input, reasoning and output. A cheaper model may need more retries or more human correction. A premium model may produce a stronger answer that adds no value because the task is simple. A fast response may be commercially valuable in a live customer interaction but irrelevant in an overnight process.

Public benchmarks are useful for understanding broad capability. They do not reproduce your data, instructions, integrations, edge cases, users or consequences. Vendor demonstrations are narrower still: they show what can work under selected conditions.

The appropriate unit of comparison is therefore the accepted result in a defined workflow.

Start with a cost ladder:

1.  Cost per attempt: model usage and any tool or data calls for one run.

2.  Cost per accepted task: attempts, retries and human review divided by outputs that meet the agreed standard.

3.  Cost per business outcome: accepted tasks, operating costs and downstream effects compared with the commercial result the workflow exists to improve.

Not every pilot can establish the third measure immediately. A proposal assistant might first be judged on accepted drafts, preparation time and error rate before the business has enough evidence to assess conversion or margin. The important point is to design the line of sight from technical output to business outcome at the start.

Define success before naming the models

Model selection should begin with the work, not the supplier list. Describe one workflow precisely: the user, input, action, expected output, human decision, consequence of error and current baseline.

Then define an acceptance threshold. For a routine classification task, it may be a measurable accuracy rate and response time. For a judgement-heavy task, it may be an expert rubric covering factual accuracy, use of evidence, completeness, appropriate uncertainty and actionability.

Both OpenAI’s evaluation guidance and Anthropic’s guidance on defining success and building evaluations stress task-specific tests, explicit criteria and realistic cases. They also treat evaluation as multidimensional. Quality, consistency, latency, privacy and price may all matter, but their importance changes by use case.

This is why a single enterprise-wide model score is rarely helpful. A customer complaint, contract summary, product description and credit decision do not have the same standard or cost of failure.

Build a business-weighted scorecard

Use a small number of criteria and weight them before seeing the results. Otherwise, teams tend to favour whichever output looks most fluent.

Outcome quality. Does the output meet the task standard? Test accuracy, completeness, evidence and usefulness to the next person in the workflow.

Reliability. Does performance hold across ordinary, incomplete, contradictory and unusual inputs? Record variability, refusals and fabricated or unsupported claims.

Human effort. How much time is spent preparing input, checking, correcting and recovering failures? Human review is not free, and an apparently inexpensive model can become costly if experienced staff repeatedly repair its work.

Operating performance. Measure end-to-end response time, availability, rate limits and integration behaviour. Output speed alone does not describe the user’s experience.

Risk and control. Consider data handling, access, auditability, security, explainability, supplier jurisdiction and the consequence of a wrong answer. Higher-consequence tasks require stronger evidence and oversight.

Total cost and portability. Include usage, licences, integration, support, monitoring and change. Assess how difficult it would be to move the workflow to another model or supplier.

The scorecard should produce a decision, not a beauty contest. A model passes if it clears the required thresholds at an acceptable total cost. Extra capability only earns its place when it changes the outcome.

Run a fair workflow test

A credible comparison needs equivalent conditions. Give each candidate the same task definition, source information, instructions, output format and tool access. Where platforms have different integration options, document the difference rather than pretending the test is symmetrical.

Use a test set drawn from real or carefully sanitised work. Include normal cases, difficult cases and failures from the existing process. Ask domain experts to define reference answers or scoring rubrics. Blind the reviewer to the model where practical.

For each run, log:

• whether the output was accepted, corrected, retried or rejected;
• the reason it failed the standard;
• usage and end-to-end latency;
• review and correction time;
• any integration, permission or data issue;
• the final outcome delivered to the user or next process step.

Do not tune one candidate repeatedly while giving another its first attempt. Allow a proportionate configuration period, then freeze the test conditions. The goal is not a laboratory claim that one provider is universally superior. It is evidence about how complete systems perform in your operating context.

Evaluation must continue after deployment. The NIST AI Risk Management Framework treats governance, measurement and management as lifecycle activities. Models change, suppliers update services, business inputs drift and users find new ways to interact with the workflow. A decision made at launch can expire.

Route work instead of declaring one winner

A well-designed architecture can allocate different work to different models or settings. The routing policy should reflect complexity, consequence and confidence rather than departmental preference.

A simple pattern is:

• route predictable, low-consequence, high-volume work to the lowest-cost option that consistently clears the acceptance threshold;
• escalate complex, ambiguous or higher-value cases to a more capable model or greater reasoning setting;
• require human review, a specialist route or no automation where the consequence exceeds the organisation’s tolerance;
• retain a fallback for supplier outage, performance deterioration or a material price change.

Routing can reduce cost, but it adds engineering, monitoring and governance overhead. For an SME with a small number of workloads, one well-tested provider may be more economical than a sophisticated multi-model layer. Complexity must also earn its place.

Procure for change, not permanence

The UK Government’s guidelines for AI procurement recommend a clear problem statement, whole-life cost consideration, ongoing evaluation and measures to reduce vendor lock-in. Although written for public procurement, those principles are valuable for established SMEs buying AI-enabled software and services.

Before signing, clarify:

• which models and versions support the service and how changes are communicated;
• how business data, prompts and outputs are stored, used and deleted;
• what performance evidence and incident information the supplier will provide;
• how usage, support and exit costs can change;
• whether prompts, evaluation sets, workflow logic and business data can move elsewhere;
• what happens if a model is withdrawn or no longer meets the threshold.

Do not demand complete technical disclosure where it is unrealistic. Do demand enough transparency and portability to remain accountable for the business process.

A practical next step

Select one material AI-enabled workflow and calculate cost per accepted task for the last month. Include retries and review time, not just the invoice. Build a 20-to-50-case evaluation set from real work, agree the pass threshold with the process owner and test one credible alternative under equivalent conditions.

If model and supplier choices are growing faster than the controls around them, explore The AI Centre’s AI governance and risk support. The objective is a repeatable decision system that keeps models accountable to the work.

Sources

• Evaluation best practices, OpenAI, accessed 12 October 2026.
• Define success criteria and build evaluations, Anthropic, accessed 12 October 2026.
• AI Risk Management Framework, National Institute of Standards and Technology, released 26 January 2023.
• Guidelines for AI procurement, UK Government, 8 June 2020.