How to Evaluate Intent Recognition: From Classification Accuracy to Multi-turn Task Success Rate

An intent recognizer's accuracy going from 90% to 93% doesn't necessarily mean the assistant has become more useful. The additional correct predictions might all come from high-frequency queries, while the model still frequently misjudges order cancellations. Alternatively, the model may have become more conservative, routing numerous requests for clarification, which naturally raises the accuracy of accepted requests.

Evaluation needs to answer a more specific question: What capability did this change improve, what was the cost, and did it make it easier for users to accomplish their tasks? This article introduces evaluation methods for classification, slots, dialogue state, and business results, and provides ready-to-run evaluation scripts. Download the accompanying project.

1. Evaluation Targets and Metrics

A hotel assistant can be broken down into four stages: understanding user input, updating dialogue state, selecting the handling path, and executing the business process. Each stage has different correct answers, so a single "accuracy" metric can't cover everything.

When a user says "change check-in to tomorrow," intent recognition might classify it as continuing the booking—that's the classification layer's answer. The state update also needs to preserve city and room type while only changing the check-in date—that's the state layer's answer. The final booking still needs to satisfy date, availability, price, and confirmation requirements—that's the task layer's answer.

If we only look at whether the room was finally booked, it's hard to know where the failure happened when it does occur. If we only check whether labels are correct, we might miss state overwrites, dropped conditions, and tool execution errors. Reasonable evaluation preserves both layered diagnostics and end-to-end results.

2. Splitting Training, Validation, and Test Sets

The training set is used for learning parameters, the validation set for selecting hyperparameters and thresholds, and the test set for checking results after the approach is finalized. The difference between them is how they participate in decision-making, not just the file names.

Suppose one original dialogue generates five paraphrases. Randomly distributing six records into training and test means the model might see nearly identical expressions in the test set. Grouping relationships like dialogues, paraphrase families, users, or sources should form clusters first, then be split.

For example, "I want to book a hotel in Beijing" and the subsequent "change to tomorrow" from the same session cannot be placed in training and test separately, otherwise the model won't be tested on understanding new dialogues. The context needed for testing must be preserved with the entire conversation.

In practice, you can add conversation_id, source_id, and family_id to each record, then choose the appropriate grouping key based on the task. For already-deployed systems, you can also use time-based splitting to observe new expressions: train on earlier data, test on data from a subsequent period. Time-based splitting also needs to prevent the same ticket or session from crossing the boundary.

Dynamic example retrieval is especially easy to overlook. Even if the classifier hasn't trained on test sentences, if the Few-shot example library or nearest-neighbor index contains test answers, the entire system is still using test information. The retrieval database, prompt examples, thresholds, and model are all part of the system being evaluated.

3. Accuracy and Macro-F1

Accuracy is correct predictions divided by total predictions—intuitive, but affected by class frequency.

Suppose there are 100 requests, with 90 order queries and 10 cancellation requests. If the system predicts all as queries, the accuracy is still 90%, but the cancellation class recall is 0. For the cancellation flow, this system provides almost no detection capability.

For a given class, treat the predictions as two sets: "belongs to this class" and "does not belong to this class":

Precision = TP / (TP + FP)
Recall    = TP / (TP + FN)
F1        = 2 × Precision × Recall / (Precision + Recall)
Macro-F1  = arithmetic mean of F1 across all classes

Precision tells you how many selected items are actually correct, and recall tells you how many of the actually existing items were retrieved. F1 considers both; Macro-F1 gives each class equal weight, making it easier to expose problems with low-frequency classes. Calculation requires a fixed label list and clarity on how to handle cases with no predictions or no supporting samples. scikit-learn F1 documentation

In the 90/10 example above, the query class F1 is approximately 0.947, the cancellation class is 0, and Macro-F1 is approximately 0.474. These numbers reveal the class bias that accuracy hides.

But Macro-F1 also doesn't automatically express business cost. Misclassifying a query as cancellation, and misclassifying a refund inquiry as order query might have different consequences. High-cost classes should still have separate recall, false acceptance, and business process outcomes tracked.

4. Using Confusion Matrix to Analyze Errors

The confusion matrix rows represent true classes, and columns represent predicted classes. The counts on the diagonal are correct classifications; the other cells tell us where errors flow.

For example, modifying existing orders often falls into room queries, which means the model might only have caught the keyword "room type" while ignoring the "already booked" stage information. Querying cancellation rules often falls into cancellation actions, which requires checking positive and negative examples along with question patterns, conditions, and negation expressions.

The previous baseline method's predictions can be recalculated using batch3/evaluate.py. Here, the raw classification metrics only count the 14 known-category inputs, while rejection metrics separately use all 21 inputs:

This table does not treat rejection as correct classification. Candidate ranking and acceptance are two separate decisions and should be checked separately; otherwise, you can't tell if a scheme is better at classification or just accepting easier requests.

5. Coverage and Rejection Effects

Consider two strategies: A accepts 100 items with 5 errors; B accepts 10 items with 0 errors. If you only look at acceptance accuracy, B is higher. But the remaining 90 items still need to be processed, and the system might cause many users to repeatedly explain their needs.

Therefore, report at least:

Coverage = number accepted and labeled / total inputs
False acceptance rate = number of erroneous acceptances / number accepted
Out-of-scope misacceptance rate = out-of-scope items accepted / total out-of-scope items

When no samples are accepted, the false acceptance rate has no denominator and should be marked as undefined. Don't write 0% and then treat it as an excellent result.

Threshold scanning can plot the relationship between coverage and error rate, but it shouldn't be assumed to be smooth or monotonic. With finite samples, removing a few correct or erroneous results can make the ratio jump. Choose thresholds based on validation data and error costs, fix them, then check test performance; avoid repeatedly adjusting thresholds based on test results.

6. Multi-label, Slot, and State Evaluation

For multi-label tasks, treat results as sets. Predicting {query order} when the true value is {query order, refund inquiry} is a miss; predicting an additional cancellation task is an over-prediction. You can calculate set precision, recall, or require exact set matching.

But two booking tasks might share the same label. Comparing only label sets would merge them. Therefore, task matching should combine task identity, key parameters, and dependencies—first matching predicted nodes to ground truth nodes when necessary, then checking relationships.

Slot matching also requires business semantics. Whether "tomorrow" equals "2026-10-02" depends on conversation time and timezone; whether "北京市" equals "北京" depends on normalization rules. Normalization must be applied uniformly to predictions and ground truth, not added as special rules to fix a specific result.

Full state matching checks whether all evaluated fields are correct after this turn. It's stricter than averaging per-slot accuracy: if city, date, or room type is wrong, the entire state doesn't match. Looking at both metrics together can distinguish local field improvements from complete state usability.

7. Success Criteria for Multi-turn Tasks

"The assistant said it's already cancelled" is not evidence of cancellation success. Success criteria should correspond to tool results or business state—for example, the order status actually becoming cancelled, with no duplicate actions generated.

Similarly, when a user says "don't cancel if there's a charge," and upon checking there's a fee so the order is preserved, this should count as honoring the goal, not simply counted as cancellation failure. Multi-turn evaluation needs to clearly state the allowed final states for each scenario in advance.

For a conditional cancellation scenario, you can save: initial order, user constraints, allowed actions, expected final state, and prohibited actions. After executing the dialogue, check both the result and the process: whether necessary information was requested, whether the fee was displayed, whether it was incorrectly cancelled, and whether it was submitted multiple times.

Clarification also has costs. Beyond task completion rate, also record average clarification count, repeated clarifications for the same gap, and where users abandon the flow. A flow that ultimately succeeds but requires ten repeated explanations still has clear room for improvement.

8. Measuring Latency and Cost

Model inference is only part of end-to-end latency. What users experience also includes queuing, state retrieval, retrieval, tool calls, and retries.

When measuring, first define the start and end points: from when the HTTP request arrives until the final response or explicit clarification result is returned. Cache hits, model downgrades, failed requests, and timeouts should all be preserved—don't only average the fast requests that succeeded.

Averages are convenient for estimating total resources; P95 and other percentiles help observe tail experience. When comparing two implementations, also keep concurrency, input length, hardware, and warm-up method consistent. Cost per completed task can count retries and additional calls when escalating to larger models.

This series' local project records functional checks, model classification experiments, and service request tests separately. The three answer different questions: whether the logic is correct, how the model classifies, and whether the request chain handles contracts properly.

9. Improving the System Based on Evaluation Results

First group errors: label definition conflicts, expression coverage gaps, missing context, parameter binding errors, overly conservative thresholds, routing errors, and tool failures. Then propose changes for one type of error and observe corresponding metrics as well as potentially affected other metrics.

For example, supplementing cancellation policy counterexamples should focus on confusion between queries and cancellations; changing state summarization should focus on pronouns and short replies; raising thresholds should look at both false acceptance and coverage.

Multiple random seeds help observe the impact of training initialization, but they can't replace test data from more sources. When resampling multi-turn data, sample by conversation rather than treating each turn in the same dialogue as independent samples. When using models to judge answers, also first verify the judging criteria against human-checked samples to avoid the judge just favoring familiar phrasing.

Run this evaluation:

python batch3/evaluate.py

Results are saved in batch3/results/hotel-evaluation.json, including per-class metrics, confusion matrices, and threshold scans at fixed split differences. For next steps, read Small Model Training and Distillation to learn how to determine training objectives based on issues found in evaluation.