A fluent AI answer can still be wrong.

That is one of the most important things to understand about large language models.

An answer can be clear.

Confident.

Well-structured.

Persuasive.

And still contain a factual error, misunderstand the task, ignore an instruction, invent a source or fail in a way that is difficult to notice at first glance.

This is why one of the most important questions in AI engineering is not:

“Does the model sound intelligent?”

It is:

“How do we know the system is actually working?”

That is the purpose of LLM evaluation, often shortened to evals.

Evals are structured ways of testing the performance of a large language model or an application built around one.

They help us move away from:

“I tried it a few times and it seemed good.”

toward:

“We defined what good performance means, tested the system against that definition and measured the results.”

That difference matters enormously once AI moves from experimentation into real products.

What Is LLM Evaluation?

LLM evaluation is the process of systematically testing how well a language model or LLM-powered system performs on the tasks we care about.

That can include questions such as:

Does it answer correctly?

Does it follow instructions?

Does it invent information?

Does it use retrieved evidence properly?

Does it remain reliable when the wording of a request changes?

Does it refuse when it should?

Does it complete the task within an acceptable amount of time?

How much does each successful interaction cost?

OpenAI's current evaluation guidance describes evals as structured tests used to measure model performance despite the variability of generative AI systems. It recommends task-specific evaluations, early and repeated testing, logging, automated scoring where appropriate and calibration against human judgment.

The important point is that evaluation is not one score.

It is a measurement system.

Why Testing LLMs Is Different From Testing Ordinary Software

Traditional software is often deterministic.

If I write:

2 + 2

I expect:

4

every time.

If the software returns 5, something is wrong.

Large language models behave differently.

Their outputs can vary.

Two requests that appear nearly identical may produce different answers.

The same model may generate different wording or reasoning for the same task.

And a response can be partially correct rather than simply right or wrong.

This does not make ordinary software testing useless.

It means we need additional evaluation methods for probabilistic systems.

OpenAI's evaluation guidance explicitly notes that because generative AI is variable, traditional software tests alone are not sufficient for many LLM architectures.

That is why testing an AI product requires both:

deterministic tests where deterministic behaviour is expected

and

probabilistic evaluations where model quality must be measured.

A Good Answer Is Not the Same as a Correct Answer

One of the most dangerous properties of modern language models is that incorrect answers can still sound convincing.

NIST uses the term confabulation for cases where generative AI confidently produces erroneous or false content, including outputs that contradict the input or previously generated information. The same phenomenon is commonly called hallucination.

The problem is not only that the model can be wrong.

Humans can be wrong too.

The problem is that linguistic confidence is not a reliable measure of factual confidence.

Consider:

“What year did Cameroon launch its National Artificial Intelligence Strategy?”

A system might respond confidently with a specific year.

If we judge the answer by writing style, we learn almost nothing.

We need to compare the answer against reliable evidence.

That difference between fluency and correctness is one of the central reasons evals matter.

What Should We Evaluate?

There is no universal evaluation score that tells us whether an AI system is “good.”

The correct metrics depend on the job we expect the system to perform.

Stanford's HELM project was created partly around this problem: language models are general-purpose systems, so evaluation should cover multiple scenarios and multiple dimensions rather than relying on one accuracy number. Its evaluation frameworks include dimensions such as capability, robustness, fairness, multilingual performance and efficiency.

For a real AI product, I would think about evaluation across several major dimensions.

Accuracy and task success

The most obvious question is:

Did the system accomplish the task correctly?

If an AI system extracts invoice totals, we can compare the extracted value with the correct value.

If it classifies customer requests, we can compare its classification with labelled examples.

If it answers factual questions, we can compare its claims with verified evidence.

The exact metric depends on the task.

Sometimes accuracy works.

Sometimes precision, recall or F1 score may be more appropriate.

Sometimes the correct measure is simply:

Did the user successfully complete the task?

The metric should follow the purpose of the product, not the other way around.

Hallucination and Groundedness

For systems that generate factual information, we also need to know:

Did the system invent anything?

This becomes particularly important in systems using retrieval-augmented generation, or RAG.

Imagine an AI system that searches company documents before answering.

It gives a correct-looking answer and cites three documents.

That does not automatically mean the system worked correctly.

We need to evaluate:

Did it retrieve the relevant document?

Did the answer actually follow from that document?

Did the citation support the claim?

Did the model add unsupported information?

These are different problems.

A useful evaluation can therefore separate:

retrieval quality

from

answer quality

from

citation correctness.

This becomes especially important in high-stakes domains where an unsupported but plausible sentence can be costly.

Instruction Following

Sometimes an answer can be factually correct and still fail the task.

Suppose the user says:

Summarize this document in five bullet points and do not include recommendations.

The system writes an excellent summary.

But it uses twelve paragraphs and ends with three recommendations.

The information may be correct.

The task was still not completed correctly.

Instruction-following evals measure whether the model respects constraints such as:

format,

length,

tone,

required information,

forbidden information,

and ordering.

Stanford's HELM Instruct framework, for example, evaluates instruction-following models across multiple criteria rather than treating response quality as one undifferentiated score.

Robustness

Another important question is:

Does the system continue to perform well when the input changes slightly?

Imagine an AI that performs perfectly when asked:

What is your refund policy?

But fails when someone writes:

I paid for something and I don't want it anymore. Can I get my money back?

The underlying intent is similar.

A reliable system should not depend on users phrasing every request exactly like the test examples.

Robustness testing can include:

misspellings,

longer prompts,

short prompts,

ambiguous wording,

code-switching,

different languages,

irrelevant information,

or adversarial input.

For a system intended for Cameroon, for example, we might test not only formal English and French, but also conversational phrasing, code-switching and other real input patterns relevant to the target users.

That is one reason local evaluation matters.

Reliability and Consistency

Reliability asks:

How often does the system perform acceptably?

A model that produces a brilliant result nine times out of ten but fails catastrophically on the tenth attempt may be unsuitable for some applications.

A creative writing application can tolerate more variation than a system extracting financial figures.

The acceptable error rate depends on the consequences of failure.

This is why evaluation should reflect the actual risk of the use case.

The question is not merely:

What percentage did the model score?

It is:

Is this level and type of error acceptable for this application?

Safety

We may also need to evaluate whether the system behaves safely.

That can include whether it:

reveals sensitive information,

follows malicious instructions,

produces prohibited content,

misuses tools,

ignores permission boundaries,

or responds appropriately when uncertain.

Stanford's 2026 AI Index notes that responsible-AI benchmarking is increasing but still lags behind the pace of AI capabilities and deployment. It also highlights significant variation in model performance on tests designed to measure hallucination and reliability-related behaviour.

That does not mean one benchmark can tell us whether a system is safe.

It means safety itself has to be tested rather than assumed.

Latency Is an Evaluation Metric Too

AI quality is not only about correctness.

Imagine two systems.

System A is slightly more accurate but takes 25 seconds to respond.

System B is slightly less accurate but responds in two seconds.

Which is better?

Again, it depends on the use case.

For an AI system helping someone conduct deep research, 25 seconds may be acceptable.

For an autocomplete system or customer-service assistant, it may not be.

This means evaluation should often include:

response latency

alongside quality.

Stanford's recent AI Index work notes that as top-model capability converges in some evaluations, factors such as cost, reliability and domain-specific performance become increasingly important differentiators.

Cost Is Part of Performance

Cost deserves the same treatment.

Suppose Model A achieves:

94% task success

at a cost of:

$100 per 10,000 requests.

Model B achieves:

92% task success

at:

$12 per 10,000 requests.

Which one should we deploy?

The answer depends on how much those two percentage points matter.

For a low-risk support task, Model B may be the better engineering decision.

For a high-value task where each failure costs thousands of dollars, Model A may be preferable.

This is why the question:

Which model is best?

is often badly framed.

A better question is:

Which model provides the best performance for this task under our quality, latency, risk and cost constraints?

That is a much more useful engineering question.

Benchmarks Are Useful, but They Are Not Your Product

Public benchmarks are important.

Tests such as MMLU, SWE-bench and other standardized evaluations help researchers and developers compare model capabilities.

Stanford's HELM project, for example, publishes reproducible evaluations across many models, scenarios and metrics.

But a model scoring highly on a public benchmark does not automatically mean it will perform well inside your application.

Suppose you are building an AI system for:

Cameroonian regulatory documents,

LPG consumption forecasting explanations,

customer complaints,

insurance claims,

or bilingual administrative workflows.

A general benchmark may tell you something about the model.

It cannot tell you everything about your product.

This is why production AI systems need task-specific evals.

Your evaluation dataset should resemble the problems your users will actually give the system.

The Test Set Should Look Like Reality

One of the easiest ways to create misleading evaluation results is to test only easy examples.

Suppose an AI customer-support system is tested on:

Where is my order?

How do I reset my password?

What are your opening hours?

It performs brilliantly.

Then real customers arrive with:

I was charged twice, one order was cancelled, the refund hasn't arrived and your agent told me something different yesterday.

The evaluation did not fail because the model changed.

The evaluation failed because the test set did not represent reality.

Good eval datasets should contain:

normal cases,

difficult cases,

edge cases,

past failures,

ambiguous requests,

and cases that are particularly important to the business.

OpenAI's evaluation guidance specifically recommends building task-specific evaluations that resemble real production distributions and using logs from actual application behaviour to improve the test set over time.

This is where evaluation becomes a continuous engineering process rather than something performed once before launch.

Human Evaluation Still Matters

Not every quality dimension can be measured with an exact formula.

Consider:

Is this explanation easy for a beginner to understand?

Is this response professional?

Does this answer fully address the user's concern?

Those involve judgment.

Human reviewers can score responses using a clear rubric.

For example:

CriterionQuestionCorrectnessAre the factual claims accurate?CompletenessDid the answer address the full task?GroundednessAre factual claims supported by the supplied evidence?Instruction followingDid the system respect the user's constraints?ClarityIs the response understandable for the target audience?SafetyDid the response remain within the system's required boundaries?

The important word is rubric.

Simply asking reviewers:

“Do you like this answer?”

creates a weak evaluation.

Reviewers need clearly defined criteria and examples of what different performance levels mean.

OpenAI's current guidance recommends calibrating automated evaluations against human feedback rather than treating automated scoring as unquestionable ground truth.

Can AI Evaluate AI?

Yes.

But carefully.

One increasingly common technique is called LLM-as-a-judge.

A strong language model is given:

the task,

the system's response,

possibly a reference answer,

and an evaluation rubric.

It then scores or compares the response.

This is useful because human evaluation can be expensive and slow.

A model judge can evaluate thousands of responses much faster.

Current evaluation platforms support model-based grading alongside methods such as exact string checks, similarity scoring and code-based evaluation.

But there is an obvious problem:

the evaluator is also an AI model.

It can make mistakes.

It can have biases.

Its scoring can vary.

This is why model-based evaluation should itself be evaluated.

A sensible process is:

have humans label a representative sample,

run the AI judge on the same sample,

compare the judgments,

and verify whether the automated grader agrees sufficiently with human expectations.

If it does not, the grading rubric or judge may need improvement.

The Evaluation Method Should Match the Task

Different tasks require different evaluation methods.

If the expected answer is:

42

exact match may work perfectly.

If the system generates a paragraph explaining a medical concept, exact matching would be absurd.

If it generates Python code, we may actually run tests against the code.

If it summarizes a document, we might combine factual checks, coverage measures and human review.

If it retrieves documents, we might measure whether the correct documents appeared.

If it uses tools, we might inspect whether the correct tool was chosen with the correct arguments.

There is no single “LLM evaluation metric.”

There is only:

the metric appropriate to the behaviour we are trying to measure.

Evaluating RAG Systems

Retrieval-augmented generation introduces an additional evaluation problem.

A RAG system has at least two major stages:

retrieval

and

generation.

If the final answer is wrong, we need to know why.

Did the system retrieve the wrong documents?

Or did it retrieve the correct documents and then misinterpret them?

Those are different failures requiring different fixes.

A strong RAG evaluation might therefore examine:

retrieval relevance,

retrieval coverage,

answer correctness,

groundedness,

citation accuracy,

and refusal behaviour when the evidence is insufficient.

Without separating these components, teams can spend time changing the model when the real problem is retrieval.

Evaluating AI Agents

Agents create an even more complex problem.

An ordinary LLM might produce one response.

An agent might:

interpret the goal,

call a search tool,

read the results,

call a database,

make a decision,

perform an action,

and then report back.

Evaluation must therefore move beyond response quality.

We may need to measure:

Did the agent accomplish the goal?

Did it select the correct tools?

Did it pass correct arguments?

Did it take unnecessary actions?

Did it respect permission boundaries?

Did it recover from failures?

Did it stop at the right point?

Did it ask for human approval when required?

This is why agent evaluation becomes a system evaluation, not simply a language evaluation.

Evaluation Should Begin Before Launch

One of the weakest AI development patterns is:

build → launch → discover problems → then think about evaluation.

A stronger approach is:

define success → build a small evaluation set → develop → run evals → improve → expand the evaluation set → deploy → continue monitoring.

OpenAI's current guidance refers to this as eval-driven development: evaluation should happen early and repeatedly, not as a final quality-control step.

This changes development behaviour.

Every major modification becomes testable.

Change the prompt?

Run the evals.

Change the model?

Run the evals.

Add RAG?

Run the evals.

Change chunk size?

Run the evals.

Add a tool?

Run the evals.

Reduce cost?

Run the evals.

The question becomes:

Did this change make the system better, worse or simply different?

Without measurement, we are guessing.

Regression Testing Matters in AI Too

Suppose your AI system performs well today.

You improve the prompt.

Ten previously weak cases become better.

Excellent.

But perhaps five cases that were previously correct are now worse.

This is a regression.

Software engineers have dealt with regression testing for decades.

AI systems need the same discipline.

A good evaluation suite becomes a permanent collection of tasks the system must continue to pass.

Every important failure can eventually become a new test.

That creates a powerful feedback loop:

production failure → understand cause → add eval case → fix system → prevent recurrence.

Over time, the evaluation suite becomes a record of what the system has learned not to break.

Do Not Optimize for the Evaluation Instead of the User

There is another danger.

Once we create a metric, teams naturally try to improve it.

But models can sometimes improve on a benchmark without becoming meaningfully better for users.

This is one reason Stanford's 2026 AI Index warns that AI capabilities can progress faster than the benchmarks designed to measure them, with some evaluations becoming saturated quickly.

A benchmark is therefore a measurement tool.

It is not the objective itself.

If users care about accurate answers in French and English, but the team spends months optimizing an unrelated English reasoning benchmark, the evaluation programme has lost contact with the product.

Measure what matters.

Not simply what is easiest to measure.

Common Evaluation Mistakes

There are several ways teams can fool themselves into thinking an AI system is better than it is.

The most common is vibe-based evaluation:

“I tried it and it seems really good.”

That can be useful during early experimentation.

It is not enough for deployment.

Another mistake is measuring only average performance.

Imagine a system that achieves 98% accuracy.

That sounds excellent.

But what if the remaining 2% contains every case involving high-value financial transactions?

Average performance hides risk distribution.

Another mistake is evaluating only the happy path.

Another is relying on a single benchmark.

Another is creating test examples that are too similar to the examples used during development.

And another is allowing the system to improve the metric without asking whether users actually experience better outcomes.

Strong evaluation requires resisting the temptation to reduce a complex system to one flattering number.

What Does a Good LLM Evaluation Process Look Like?

Imagine we are building an AI assistant for a company.

Before launch, we define the system's responsibilities.

Then we identify the behaviours that matter:

correct answers,

appropriate refusal,

instruction following,

document grounding,

response time,

and operating cost.

We collect representative examples.

We include routine requests.

We include difficult requests.

We include failures observed during development.

We define what correct performance looks like.

Some cases can be evaluated automatically.

Others require human review.

Some may use model-based graders that have first been calibrated against humans.

Then we create a baseline.

Perhaps:

82% task success

7% unsupported-claim rate

3.2-second median latency

$0.014 average cost per request

Those figures are hypothetical examples, not universal targets.

The point is that now we have something measurable.

We change the system.

Then we run the same evaluation again.

Now engineering decisions can be based on evidence.

The Best Model Is Not Always the Best System

This leads to a broader lesson.

AI applications are systems.

The model is one component.

A slightly weaker model combined with:

better retrieval,

better tools,

better prompts,

better validation,

better caching,

and better workflow design

may outperform a theoretically stronger model placed inside a poorly designed application.

That is why evaluating only the underlying model is not enough.

We need to evaluate the whole system the user actually experiences.

Evaluation Is How AI Engineering Becomes Engineering

It is easy to build something that appears intelligent.

Connect an API.

Write a prompt.

Add an interface.

Ask it several questions.

Watch it respond.

That can be an exciting prototype.

But production engineering begins when we start asking harder questions.

How accurate is it?

Where does it fail?

How frequently does it fail?

Which users experience those failures?

How expensive is each successful task?

How fast is it?

Does a new model actually improve anything?

Can we detect regressions?

What happens on inputs we did not expect?

Those questions transform AI development from demonstration into engineering.

Because the standard should not be:

“The AI generated an impressive answer.”

The standard should be:

“We know what this system is supposed to do, we know how we measure it, and we know where it still fails.”

That is what evaluation gives us.

Not certainty.

Not perfection.

But evidence.

And in AI engineering, evidence is far more useful than confidence.

Frequently Asked Questions

What is LLM evaluation?

LLM evaluation is the structured process of measuring how well a large language model or LLM-powered application performs on defined tasks. Depending on the use case, evaluation can measure accuracy, factuality, instruction following, reliability, safety, latency, cost and other characteristics.

What are AI evals?

“Evals” is shorthand for evaluations. They can refer to public benchmarks used to compare models or to custom tests created specifically for an AI application.

How do you measure hallucinations in an LLM?

Hallucination evaluation usually requires checking whether factual claims are supported by a trusted source or supplied context. In RAG systems, this may involve measuring whether the generated answer is grounded in the retrieved documents.

Can an LLM evaluate another LLM?

Yes. This approach is commonly called LLM-as-a-judge or model-based grading. It can scale evaluation, particularly for subjective qualities, but should be validated against human judgment because model graders can also make mistakes.

Why are benchmarks not enough?

Public benchmarks measure specific capabilities under defined conditions. A model that performs well on them may still perform poorly on a company's actual data, workflows, languages or user behaviour. Production systems therefore need task-specific evaluation.

Should cost and latency be part of LLM evaluation?

Often, yes. The most accurate model may not be the best system if it is too expensive or too slow for the intended application. Production evaluation should reflect the product's real constraints.