An intent recognizer’s accuracy improving from 90% to 93% does not necessarily mean the assistant has become more useful. The additional correct predictions may all come from frequently asked questions, while order cancellations are still frequently misclassified. Alternatively, the model may have become more conservative, routing many requests to clarification and thereby improving accuracy among the requests it accepts.
Evaluation needs to answer a more specific question: which capability did this change improve, at what cost, and did it make it easier for users to complete their tasks? This article covers methods for evaluating classification, slots, dialogue state, and business outcomes, and includes an evaluation script you can run as is. Download the companion project.
I. Evaluation Targets and Metrics
A hotel assistant can be broken down into four stages: understanding what the user wants, updating the dialogue state, choosing a handling path, and executing the business workflow. The correct answer differs at each stage, so a single “accuracy” metric cannot cover every failure mode.
If a user says, “Change my check-in to tomorrow,” intent recognition might classify it as continuing the booking; that is the classification-layer answer. State updating must preserve the city and room type while changing only the check-in date; that is the state-layer answer. The final booking must satisfy the date, availability, price, and confirmation requirements; that is the task-layer answer.
If you look only at whether the final booking succeeded, a failure makes it difficult to determine what went wrong. Looking only at label correctness, on the other hand, can miss cases where state was overwritten, constraints were dropped, or tools executed incorrectly. A sound evaluation should preserve both layered diagnostics and end-to-end outcomes.
| Layer | Typical issue | Metric or check |
|---|---|---|
| Classification | Whether policy inquiries are misclassified as actions | Per-class recall, Macro-F1, confusion matrix |
| Out-of-scope rejection | Whether unsupported requests are forcibly accepted | Coverage, acceptance error rate, out-of-scope false acceptance rate |
| Arguments and state | Whether date changes apply to the correct task | Slot matching, exact state match, task binding |
| Multi-turn tasks | Whether the user’s goal is actually achieved | Completion rate, number of clarification turns, recovery success rate |
| Business | Whether incorrect or duplicate actions are triggered | Misoperations, duplicate executions, handling cost |
II. Splitting the Training, Validation, and Test Sets
The training set is used to learn the parameters, the validation set to select hyperparameters and thresholds, and the test set to check the results after the design has been fixed. What distinguishes them is how they inform decisions, not merely their file names.
Suppose a single original dialogue generated five paraphrases. If the six records are randomly split between the training and test sets, the model may encounter nearly identical wording in the test set. Relationships such as a shared dialogue, paraphrase family, user, or source should be used to form groups before splitting.
For example, “I want to book a hotel in Beijing” and the later “change it to tomorrow” from the same session cannot be placed in the training and test sets separately; doing so would make it impossible to test the model’s ability to understand new conversations. Any context required by a test example must be preserved with the full conversation.
In practice, you can first add conversation_id, source_id, and family_id to each record, then choose the appropriate grouping key for the task. For systems already in production, a temporal split can reveal how the model handles new expressions: train on earlier data and test on a later period. Temporal splits must also prevent the same ticket or session from straddling the boundary.
Dynamic example retrieval is especially easy to overlook. Even if the classifier was not trained on a test sentence, the system as a whole still has access to test information if the Few-shot example bank or nearest-neighbor index includes the test answer. The retrieval store, prompt examples, thresholds, and model are all part of the system being evaluated.
III. Accuracy and Macro-F1
Accuracy is the number of correct predictions divided by the total number of predictions. It is intuitive, but it is affected by class frequency.
With 100 requests, 90 of which ask about order status and 10 of which request cancellation, a system that predicts “order status” for every request still achieves 90% accuracy, while recall for the cancellation class is 0. For the cancellation flow, such a system provides virtually no recognition capability.
For a given class, first treat the predictions as two sets: “belongs to the class” and “does not belong to the class”:
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)
F1 = 2 × Precision × Recall / (Precision + Recall)
Macro-F1 = 各类别 F1 的算术平均
Precision measures how many of the selected results are correct; Recall measures how many of the results that are actually present are recovered. F1 accounts for both; Macro-F1 weights every class equally, making it easier to expose problems in low-frequency classes. When calculating it, fix the label list and specify how to handle classes with no predictions or no support samples. scikit-learn F1 documentation
In the 90/10 example above, the inquiry class has an F1 score of about 0.947, the cancellation class has an F1 score of 0, and Macro-F1 is about 0.474. These numbers reveal the class bias that accuracy conceals.
Macro-F1 does not, however, automatically capture business costs. Misclassifying a policy inquiry as a cancellation request and misclassifying a refund inquiry as an order-status lookup can have different consequences. High-cost classes should still have their recall, false acceptances, and business-process outcomes reported separately.
IV. Analyzing Errors with a Confusion Matrix
The rows of a confusion matrix represent the true classes, and the columns represent the predicted classes. Counts on the diagonal are correct classifications; the remaining cells show where errors flow.
Confusion matrix for the sentence-embedding baseline on known hotel inputs
For example, if requests to modify an existing order are often classified as room searches, check whether the model latched onto the topic keyword “room type” while ignoring stage information such as “already booked.” If inquiries about cancellation policies are often classified as cancellation actions, inspect the positive and negative examples as well as questions, conditions, and negated expressions.
The predictions from the previous basic-method article can be reproduced with batch3/evaluate.py. Here, the original classification metrics cover only 14 inputs with known classes, while the rejection metrics use the full set of 21 inputs:
| Method | Raw accuracy | Macro-F1 | Accepted by strategy |
|---|---|---|---|
| Rules | 0.571 | 0.477 | 10/21 |
| TF-IDF classifier | 0.643 | 0.558 | 6/21 |
| Sentence-embedding nearest neighbor | 0.714 | 0.695 | 4/21 |
This table does not count out-of-scope rejection as a correct classification. Candidate ranking and acceptance are two separate decisions and should be evaluated separately; otherwise, it is impossible to tell whether a system classifies better or merely accepts easier requests.
V. Coverage and Out-of-Scope Rejection
Consider two policies: A accepts 100 requests and gets 5 wrong; B accepts 10 requests and gets none wrong. If you look only at accuracy among accepted requests, B performs better. But the remaining 90 requests still need to be handled, and the system may force many users to explain their needs repeatedly.
Therefore, report at least the following metrics together:
If no samples are accepted, the acceptance error rate has no denominator and should be recorded as undefined. It should not be written as 0% and then treated as an excellent result.
A threshold sweep can plot the relationship between coverage and error rate, but you should not assume it will necessarily be smooth or monotonic. In a finite sample, removing even a few correct or incorrect predictions can cause the proportions to jump. Choose a threshold based on the validation data and the cost of errors, then hold it fixed when checking test performance; avoid repeatedly tuning the threshold based on test results.
VI. Evaluating Multi-Label, Slot, and State Outputs
A multi-label task can be treated as a set. If the prediction is {查订单} and the ground truth is {查订单, 退款咨询}, the second label was missed; predicting an additional cancellation task is over-recognition. You can compute set precision and recall, or require the entire set to match exactly.
However, two booking tasks may share the same labels. Comparing only label sets would collapse them into one. Task matching should therefore combine task identity, key arguments, and dependency relationships. If necessary, first match predicted nodes to ground-truth nodes, then inspect the relationships.
Slot matching also requires business semantics. Whether “tomorrow” and “2026-10-02” are equivalent depends on the conversation time and time zone; whether “Beijing Municipality” and “Beijing” are equivalent depends on the normalization rules. Normalization must be applied consistently to predictions and ground truth; do not add special rules on the fly to fix a particular result.
Exact state matching checks whether all fields under evaluation are correct at the end of the turn. It is stricter than averaging slot-level accuracy: if only one of city, date, or room type is wrong, the entire state does not match. Viewed together, the two metrics distinguish local gains in individual fields from whole-state usability.
VII. Success Criteria for Multi-Turn Tasks
“The assistant says it has canceled the order” is not evidence that the cancellation succeeded. The success criterion should correspond to a tool result or business state—for example, that the order status actually changes to cancelled, with no duplicate actions.
Likewise, if a user says “don’t cancel it if there’s a fee,” and querying the order reveals a fee, keeping the order should count as respecting the user’s goal rather than simply marking the cancellation as a failure. Multi-turn evaluation requires you to define the allowable final states for each scenario up front.
For a conditional-cancellation scenario, you can store the initial order, the user’s constraints, permitted operations, the expected final state, and actions that must not occur. After the dialogue, check both the result and the process: Did it ask for the necessary information? Did it disclose the fee? Did it cancel incorrectly? Did it submit the request more than once?
Clarification also has a cost. In addition to task completion rate, record the average number of clarification turns, how often the same missing detail is requested repeatedly, and where users abandon the flow. A flow that ultimately succeeds but requires the user to explain themselves ten times still has clear room for improvement.
VIII. Measuring Latency and Cost
Model inference accounts for only part of the end-to-end latency. Users also experience queuing, state reads, retrieval, tool calls, and retries.
When measuring, agree on the start and end points first: from when the HTTP request arrives until the final response or an explicit clarification result is returned. Measurements should include cache hits, model fallbacks, failed requests, and timeouts; do not average only fast, successful responses.
The mean is useful for estimating total resource use, while quantiles such as P95 help reveal the tail-end experience. When comparing two implementations, keep concurrency, input length, hardware, and warm-up procedure consistent. Call cost can instead be measured per completed task, including retries and the additional calls required to escalate to a larger model.
This series’ local project records functional checks, model classification experiments, and service request tests separately. The three answer different questions: Is the logic correct? How does the model classify? Can the request chain handle requests according to the contract?
IX. Improving the System Based on Evaluation Results
Start by grouping the errors: conflicting label definitions, insufficient expression coverage, missing context, incorrect argument binding, overly conservative thresholds, routing errors, and tool failures. Then propose a change for one type of error and observe both the corresponding metric and any other metrics that might be affected.
For example, after adding negative examples for the cancellation policy, focus on confusion between policy inquiries and cancellations; after changing the state summary, focus on coreference and brief replies; after raising the threshold, examine both erroneous acceptances and coverage.
Multiple random seeds help reveal the effect of training initialization, but they cannot replace test data drawn from more varied sources. When resampling multi-turn data, sample by session rather than treating each turn in the same conversation as an independent example. When using a model to judge answers, first validate the judging criteria against human-checked samples so the judge does not simply favor wording it already knows.
Run the evaluation from this article:
python batch3/evaluate.py
The results are saved to batch3/results/hotel-evaluation.json, which includes per-class metrics, the confusion matrix, and a threshold sweep at a fixed score margin. Next, read Small-model training and distillation to learn how to use evaluation findings to define training objectives.