Enterprise AI
Model evaluation
Also known as: evals, LLM evaluation, model evals
Model evaluation is the practice of measuring how well a language model or AI system performs against defined criteria, using test cases, scoring rubrics and human or automated review. Practitioners often call the individual test suites "evals". It turns subjective impressions of AI output quality into repeatable measurements that can be tracked over time.
What it is
Model evaluation covers the methods teams use to judge AI output: accuracy, groundedness in source material, tone, safety, format compliance and task completion. It usually combines a fixed set of test inputs, an expected outcome or rubric, and a scoring method that may be human review, code-based checks or another model acting as a judge. The result is a score or pass rate that can be compared across model versions, prompts and retrieval settings.
Why it matters
Any AI feature that touches customers, from a support assistant to an on-site answer box, can degrade quietly when a model, prompt or data source changes. Evaluation gives marketing and product teams evidence before launch and a warning signal afterwards. For search and discovery work it also provides a disciplined way to test how assistants describe your brand, products and policies rather than relying on anecdote.
How it works
Teams build a dataset of representative inputs, often drawn from real queries and edge cases, then define what a good answer looks like for each one. Scoring is run automatically in a pipeline where possible, with human review reserved for judgement calls, and results are stored so changes can be compared release to release. The same approach is used for brand visibility testing, where a set of buyer questions is run repeatedly against assistants and the answers are scored for accuracy and citation.
When it applies
It applies whenever an AI system is being built, tuned or relied on for customer-facing output, and whenever you need to prove that a change improved things rather than just felt better.
Examples
- A retailer keeps 200 real support questions as an eval set and scores each assistant reply for factual accuracy and whether it links to the correct policy page.
- A B2B team runs 50 buyer questions against several AI assistants each month and records whether the brand is named, described correctly and cited.
- An in-house RAG search tool is tested for groundedness by checking whether every claim in the answer appears in the retrieved documents.
How it is measured
- Pass rate against the eval set, tracked per model or prompt version
- Groundedness or citation accuracy: share of claims traceable to source content
- Human review agreement with automated scores, to validate the judging method
- Regression count: number of previously passing cases that fail after a change
Related terms in Enterprise AI
- AI agentsAI agents are software systems that use a language model to plan and carry out multi-step tasks, rather than simply returning a block of text. They can call tools, query APIs, browse websites and write to other systems in pursuit of a goal, with varying degrees of human oversight. The term covers everything from a scripted assistant that books a meeting to a research agent that gathers sources and drafts a report.
- AI chipsAI chips are processors designed or optimised to run machine learning workloads, especially the large matrix operations behind training and inference. The category covers GPUs, tensor and neural processing units, and custom ASICs, usually paired with high-bandwidth memory and fast interconnects. They are also called AI accelerators or AI semiconductors.
- AI drug discoveryAI drug discovery is the use of machine learning and computational models to support the early stages of finding and refining new medicines. Models are applied to tasks such as predicting protein structures, identifying candidate molecules, prioritising targets and estimating properties like toxicity or binding affinity. The aim is to narrow a very large search space before expensive laboratory and clinical work begins.
- AI infrastructureAI infrastructure is the stack of hardware, networking, storage and software needed to train, fine-tune and serve AI models at scale. It spans accelerators such as GPUs, the data centres and power that house them, and the orchestration and serving layers that turn raw compute into working model endpoints. For most marketing teams it is a cost and capacity constraint they consume through APIs rather than something they build.
- AI procurementAI procurement is the process of evaluating, buying and contracting artificial intelligence tools and services, from assistants and content platforms to models accessed through an application programming interface. It covers commercial terms alongside security, data protection, accuracy and integration questions that do not arise with ordinary software. Most organisations run it as a structured review with pilots, stakeholder sign off and defined exit conditions.
- AI securityAI security is the practice of protecting AI systems, their data and the applications built on them from misuse, manipulation and leakage. It covers threats such as prompt injection, data exfiltration through model outputs, unsafe tool use by agents and compromised supply chains. It also covers the controls that keep AI features safe once they are live.
Primary research · August 2026
How ChatGPT Shortlists Software Brands
An audit across 10 categories and 60 buying questions. I recorded what ChatGPT reads, throws away and links to when a buyer asks it which software to buy, and what that decides.
- 60
- Questions asked
- 10
- Software markets
- 2,680
- Results read
- 367
- Links shown
Free download
Get the full report
35 pages · PDF · 536 KB. Enter your details and it downloads straight away.