LLM Evaluation Metrics: What to Measure and How

The metrics that matter for a business LLM are the ones tied to your task: correctness, groundedness, relevance, adherence to your rules, format validity and safety. Track speed and cost separately. Score all of it against a test set built from your own real cases, and review a sample by hand. A generic benchmark score tells you little about whether the tool will behave in your office, with your customers and your data.

This guide is for owners and operations leaders who are about to use, or already use, a chatbot, an assistant or an automated workflow built on a large language model (LLM). It explains what to measure, how to score responses, and what to do when the numbers move.

Table of Contents

• What does LLM evaluation actually measure?
• Which quality metrics should you use?
• Which operational metrics belong on the same dashboard?
• How do you evaluate responses in practice?
• How do you build a test set that represents your business?
• What should an evaluation framework include?
• How do you measure a customer-service chatbot?
• How do you monitor results after launch?
• Our perspective
• Want help setting this up for your business?
• Frequently asked questions
• Sources

What does LLM evaluation actually measure?

It measures whether the model does the job you gave it, reliably. OpenAI describes evals as structured tests of model performance and recommends five steps: define the objective, collect a dataset, define metrics, run and compare evals, and keep evaluating after every change.

That last step matters most for a small business. A model update, a new prompt or a new document in your knowledge base can all change behaviour. An evaluation is not a one-time sign-off. It is a small, repeatable check you run whenever something changes.

There are two families of measures, and mixing them up causes most confusion:

• Quality metrics describe the response: is it right, relevant, safe, in the right format?
• Operational metrics describe the system: how fast, how often it fails, what it costs.

You need both. A perfect answer that arrives in 40 seconds loses the customer. A fast answer that invents a refund policy costs you more than the delay would have.

Which quality metrics should you use?

Start with the short list. Databricks suggests task-specific correctness, groundedness, relevance, guideline adherence, format validity and safety, with tool-call correctness added if the system takes actions such as updating a record. Here is what each means in plain language, with a small-business example.

You will see other metric names in the wild, such as fluency, coherence and creativity. Fluency and coherence rarely separate good from bad on modern models, since they almost always read well. That is the trap: a fluent answer can still be wrong. Weight your effort toward correctness and groundedness.

Why accuracy and bias deserve a place on the list

University of British Columbia guidance on generative AI flags accuracy, reproducibility, veracity and bias as concerns, because LLMs can produce factual inaccuracies, fabricated citations and biased outputs. For a business, the practical translation is simple: include test cases where the right answer is "I don't know, let me connect you to a person," and check that the tool says so instead of improvising.

Let your goals pick the metrics

Singapore's IMDA, in its testing kit for LLM applications, says metrics should align with the test objectives, the dataset and the business or policy objectives. Where no reference answers exist, teams may need to write their own acceptability rules. That is useful for you too. If you have no "gold" answer for a task, write a short rule: what must the response include, what must it never include?

Which operational metrics belong on the same dashboard?

Track these separately from quality so a slow system does not hide behind good answers, or the reverse. Databricks lists tail latency, time to first byte, error and fallback rates, token usage and cost. Microsoft Research names time to first token, requests per second and streamed tokens per second and recommends looking at latency across every layer of the workflow, not only the model call.

For a small business, a workable starting set is:

1. Slowest responses, not the average. Look at the slowest 5 to 10 percent. That is what your unhappiest customer experienced.
2. Error and fallback rate. How often did the system fail or hand off to a person?
3. Cost per task. Total usage cost divided by completed tasks, such as resolved chats or processed invoices.
4. Handoff rate. The share of conversations that needed a human, and whether they should have.

This last number connects directly to return on investment. If you are trying to quantify time saved, pair these figures with the operational efficiency metrics and scorecard we use for operations teams.

How do you evaluate responses in practice?

Use three layers together: automated checks, a model acting as judge, and human review. OpenAI's guidance is to use task-specific evals built on production and domain-specific data, combine metric-based checks, human judgment and LLM judges, and calibrate automated scoring against human feedback.

Snowflake makes the underlying point well: because many different answers can be valid, you should combine metrics, executable checks, human judgements and model-based judges rather than relying only on how similar the text is to a reference answer. Datadog describes the same idea as a layered stack: deterministic checks, semantic or retrieval-grounded checks, rubric-graded scoring by a model, and human review.

Layer 1: deterministic checks

These are rules a script can verify with no judgment involved. They are cheap, fast and unambiguous.

• Does the output contain every required field?
• Is the date in the right format and in the future?
• Does the total match the sum of the line items?
• Does the reply avoid banned phrases, such as guarantees or legal promises?

Checks like these also catch the boring errors that cost real money. The same logic underlies approaches that reduce data entry errors by validating information before it reaches your systems, using AI where a rule cannot decide.

Layer 2: a model as judge

For qualities a rule cannot capture, such as "is this reply polite and does it answer the question," a second model can score responses against a written rubric. Keep the rubric short and specific, score on a small scale, and ask for a one-line reason. Then check the judge itself: have a person score 30 to 50 of the same responses and compare. If the judge and the human disagree often, fix the rubric before you trust the numbers.

Layer 3: human review

Human review is the reference point that everything else is calibrated against. Toronto Public Health's 2025 report describes a governance framework that assesses AI before deployment and requires generated content to be checked for bias and correctness, with human confirmation before it informs decisions. You do not need a public-sector process, but the principle scales down: for anything that touches money, contracts or personal information, a person confirms before it goes out.

How do you build a test set that represents your business?

Build it from your own work. Snowflake's guidance is clear that benchmark scores alone are not enough, since performance can shift when the task, data or operating conditions differ. A public leaderboard says how a model does on someone else's questions.

Here is a simple way to get to a first usable set in a week or two:

1. Pull 50 real examples. Past customer emails, support chats, quotes, invoices or scheduling requests. Remove personal details.
2. Mix difficulty. Include routine cases, messy ones, and a handful of traps: out-of-scope questions, angry customers, missing information.
3. Write the expected outcome. Not always a full answer. Sometimes a rule: "must mention the 48-hour cancellation window," or "must escalate to a person."
4. Score the first run by hand. This teaches you what "good" means for you.
5. Add every real failure. Each time the tool gets something wrong in daily use, that case joins the set. OpenAI's guidance also says to keep growing the test set with new cases.

The 2025 Practical Guide for Evaluating LLMs and LLM-Reliant Systems adds a useful warning: datasets and metrics must match the evaluation scope and objectives, and no single metric is enough for a holistic assessment. If someone shows you one headline score for a tool, ask what it was measured on.

What should an evaluation framework include?

The same guide says a framework should define scope and objectives, curate representative datasets, select metrics aligned with them, and set an evaluation methodology, and that you should choose based on your goals and real-world requirements. "Framework" sounds heavy. For a small business it can be a spreadsheet and a script. What matters is that it covers these five things:

• Scope: which task, which users, which risks.
• Test set: the cases above, stored and versioned.
• Metrics and thresholds: what score counts as passing, written down before you test.
• Method: who or what scores, how often, and on which version of the tool.
• Decision rule: what happens when the score drops.

If you are choosing between tools that offer built-in evaluation features, pick on fit with your task and your data handling needs, not on the length of the metric list. Our guide to which AI model suits which task can help you narrow the field first.

How do you measure a customer-service chatbot?

Measure whether the task was completed and whether the customer had a good experience. A systematic review of 122 task-oriented dialogue-system studies found wide variation in methods and stressed evaluating both task success and user experience, including assurance, coherence and civility. The same review argues that customer-service systems are also brand ambassadors, so a correct but cold reply is still a weak result. It recommends covering the range from intrinsic to extrinsic and manual to automatic measures, because no single measure captures the whole experience.

For a chatbot, a practical scorecard looks like this:

Microsoft Research also points to engagement signals, such as response length and the edit distance between consecutive prompts, which can show that customers are rephrasing because the first answer failed.

For a Canadian example of scale, a 2025 report describes a third-party evaluation of TELUS's bilingual customer-service application using automated and manual testing plus red-teaming across performance, robustness, safety, bias and fairness. Your version is smaller, but the shape is the same. If you serve customers in more than one language, test each language separately. For more on setting up service chat well, read our overview of AI customer service for Vancouver businesses and the comparison of AI chatbots versus hiring staff.

How do you monitor results after launch?

Re-run your test set on a schedule and whenever something changes, then watch live signals between runs. Datadog recommends using production user-experience signals as lower-cost heuristics for spotting irrelevance, negative sentiment and guardrail failures when labelling everything by hand is impractical. In practice, that means watching for:

• Thumbs-down ratings or complaints that mention the bot
• Repeated rephrasing of the same question
• A rising handoff rate
• Sudden jumps in cost per task or slow responses

When a number moves, work through the same order each time: check what changed (model version, prompt, documents), re-run the test set, find which cases fail, fix, and re-run before relying on the tool again.

The same discipline applies to switching models. Databricks advises switching only when the new candidate keeps quality on your own evaluation set while meeting acceptable latency and cost. A cheaper model that passes your test set is a win. A cheaper model you never tested is a gamble.

Keep a human in the loop for sensitive output. Toronto Public Health's approach, as described in its report, treats AI as support for human work, with structured review, protection of personal information and continuing accuracy and bias checks. If your system handles personal information, check your obligations under Canadian privacy law with a qualified adviser before you go live.

Our perspective

Most small businesses we meet don't need an evaluation lab. They need a clear answer to one question: can I trust this tool with this job on a Tuesday afternoon when nobody is watching?

We build every system around a client's own processes and data, and that is exactly why evaluation is straightforward for us. When the software is custom-built for your invoicing, scheduling and customer records, your test cases are already sitting in your business. We help you turn real examples into a test set, agree on pass marks before launch, and review results at regular milestones, so there are no surprises after go-live.

Our view is that good evaluation is also what makes AI worth using. It is how you find out which tasks you can safely hand over, and it is how the hours saved become something you can measure instead of hope for. Clients tell us they typically save 5 to 15 administrative hours a week once the system is running. We'd rather show you that on your own numbers than ask you to take our word for it.

Training matters here too. When your team knows what a good response looks like and how to flag a bad one, your test set improves every week. That is a habit, not a project, and we build it into the work.

Harry Gill

Want help setting this up for your business?

If you would rather not design an evaluation process alone, we can do it with you. We build custom software that brings your CRM, invoicing, scheduling and reporting behind one login, and we plan each project with fixed pricing, local support and regular milestones. Evaluation is part of how we decide what to automate and how we check it works.

If you already have a chatbot or assistant and are unsure how it is performing, our AI chatbots and customer service work includes setting up the checks described above. For reporting you need to trace back to source figures, see data analysis and AI reporting. To start from the question "where does AI fit in our business?", look at our AI strategy consulting and the AI readiness scorecard.

Frequently asked questions

What is the single best metric for evaluating an LLM?

There isn't one. The 2025 practical guide cited above concludes that no single metric is sufficient for a holistic assessment. For most business uses, start with correctness on your own test cases, add groundedness if the tool answers from your documents, and track latency and cost alongside.

How many test cases do I need?

Start with about 50 real examples that cover routine work and a few awkward cases. That is enough to reveal obvious problems and to teach you what good looks like. Then add every real failure you see, so the set grows with use.

Can I use an AI model to grade another AI model's answers?

Yes, with a check. A model judge works well with a short, specific rubric, but you should compare its scores with a person's on a sample of the same responses. If they disagree often, adjust the rubric before trusting the results, as OpenAI's guidance on calibrating against human feedback advises.

How often should I re-evaluate?

Re-run your test set whenever the model, prompt, documents or workflow change, and on a regular schedule such as monthly. Between runs, watch live signals like complaints, rephrased questions, handoff rate and cost per task.

Sources

• Evaluation best practices (OpenAI)
• What metrics should I use to evaluate new AI models? (Databricks)
• Starter Kit for Testing LLM-Based Applications for Safety and Reliability (Singapore IMDA)
• Using large language models and generative AI in research (UBC)
• How to Evaluate LLMs: A Complete Metric Framework (Microsoft Research)
• Building an LLM evaluation framework: best practices (Datadog)
• What Are LLM Evaluation Frameworks? (Datadog)
• LLM Evaluation: Metrics & Methods That Work (Snowflake)
• A Practical Guide for Evaluating LLMs and LLM-Reliant Systems
• Evaluating Task-oriented Dialogue Systems: A Systematic Review
• Current and Emerging Uses of Artificial Intelligence (Toronto Public Health)
• Armilla AI evaluation of TELUS customer-service application
• Reduce data entry errors

Recommended

• AI Readiness Assessment
• SMBs: 10 question, 5 pillar AI readiness scorecard in two weeks