When an intent recognizer cannot determine the category, forcibly choosing a label may lead to erroneous handling. If a user asks "issue an invoice," but the system lacks invoicing capability and routes the request to order inquiry because it detects a hotel-related term; or if a user says "cancel that," the system proceeds without confirming which order to cancel. The system needs to distinguish separately: whether the request falls outside the business scope, or whether it lacks information needed to identify the target.
This article distinguishes out-of-scope requests, interpretation ambiguity, missing parameters, and service unavailability, providing decision methods for rejection and clarification, and uses the same teaching data to actually select thresholds, observe coverage and errors. Complete code and data can be downloaded for reproduction.
1. Distinguishing Sources of Uncertainty
| Situation | Example | Reasonable Next Step |
|---|---|---|
| Target clear, missing slot | I want to book a room | Ask for specific parameters like city, dates |
| Multiple interpretations reasonable | "Cancel that," but there are two orders | Ask which order |
| Current capability not supported | Hotel assistant doesn't yet provide invoicing | Explain scope or transfer |
| Insufficient model evidence | Ambiguous input, all candidate scores are low | Request restatement or offer limited capability options |
| Service temporarily unavailable | Intent clear, but order query interface times out | Explain service issue, retry or transfer per strategy |
The last case is not unknown intent. If a tool failure is labeled as out-of-scope, subsequent error analysis cannot distinguish between understanding failure and service failure.
Out-of-scope also doesn't mean meaningless. Hotel invoicing is a reasonable request, just possibly not in the current capability catalog. CLINC treats known category classification and out-of-scope input recognition as separate issues that need individual checking, which can serve as a reference for evaluation design. CLINC paper
2. Understanding Candidate Scores
Assume the classifier only has three categories: query, booking, and cancel. Regardless of input, the maximum output value must belong to one of these. Without out-of-scope judgment or rejection strategy, the model cannot express the problem through "no suitable option."
The meaning of different scores also varies:
| Score | Can Indicate | Cannot Directly Indicate |
|---|---|---|
| Keyword hit count | How many preset conditions were hit | Correct probability of semantic judgment |
| Cosine similarity | Proximity in vector space | Probability of the request belonging to business scope |
| Classifier probability | Relative tendency under model assumptions and training categories | True correctness rate without calibration, or out-of-domain probability |
| LLM self-reported confidence | A numeric value generated by the model | Reliably calibrated probability that has been statistically calibrated |
Probability calibration studies whether prediction scores match observed event frequencies. Calibration effectiveness depends on data distribution; when the current input differs significantly from calibration data, rechecking is needed. scikit-learn calibration documentation
3. Max Score Threshold and Candidate Score Difference
One simple strategy is to first check whether the max score is sufficient, then check whether the top two scores are too close:
Running in the project root, the three results are abstain, clarify, and select respectively. These decimal values are only for testing decision branches, not recommended thresholds.
Low max score means the current model lacks sufficient support—it could be out-of-scope or just insufficient model capability. Top two scores being very close means candidates are hard to distinguish, but doesn't necessarily indicate true user semantic ambiguity. Therefore, the code returns "refuse to select candidate" and "need further differentiation," not directly judging low-score requests as out-of-scope input.
For tied candidates with identical scores, the example explicitly goes to the clarification branch to avoid label order determining the result. Non-finite values are also rejected to prevent NaN from accidentally passing after comparison.
4. Choosing Thresholds Based on Error Cost
Beyond training, this teaching experiment prepared 28 validation samples: 21 belonging to 7 known categories, 7 belonging to current out-of-scope. Search for thresholds and score differences on the validation set, then fix the selected parameters and run on another 21 demonstration check samples.
To demonstrate trade-offs, the following loss is preset:
Loss = 5 × error acceptance count + 1 × known request rejected selection count
Error acceptance includes two types: accepting out-of-scope input, or selecting a known input with the wrong label. Both rejection and clarification are considered as not selecting a label in this round; known requests thus add one to the cost, while out-of-scope requests not being selected add no cost.
This 5:1 setting expresses a preference: better to ask one more question than to reduce erroneous acceptance. Misjudging a cancellation order typically costs more than misjudging a menu query, so real businesses can set costs separately by action, while also considering the churn caused by follow-up questions and manual processing costs.
First select parameters on the validation set, then check the fixed scheme on another data set—this avoids repeatedly adjusting thresholds based on final results. Specific grids and results are in the download package.
| Method | Max Score Threshold | Score Diff Threshold | Check Set Coverage | Error Rate in Acceptance |
|---|---|---|---|---|
| Keyword rules | 0.00 | 0.00 | 47.6% (10/21) | 30.0% (3/10) |
| TF-IDF + Logistic Regression | 0.00 | 0.15 | 28.6% (6/21) | 16.7% (1/6) |
| Sentence Embedding Nearest Neighbor | 0.80 | 0.10 | 19.0% (4/21) | 0.0% (0/4) |
Setting the max score threshold to 0 doesn't mean unconditional acceptance. Rules still reject zero hits and tied max scores; TF-IDF also requires passing the candidate score difference. The two conditions should be understood together.
For example, TF-IDF only accepts 6 inputs from this set. Lower coverage means more requests enter follow-up questioning or other processing paths. When evaluating this strategy, consider both misclassification cost and subsequent processing cost, not just a single metric.
5. Limitations of Threshold Methods
High thresholds can reduce some weak match misacceptances, but they also reject correct requests that are naturally expressed but distant from training examples. If error input and correct input score distributions heavily overlap, a single threshold cannot simultaneously achieve high coverage and low error.
Similarly, candidate score difference only compares relative relationships among existing categories. An out-of-scope request may not match any category but still be obviously more like one of them; a known request may also have a small score difference due to overlapping label definitions.
When encountering these situations, check label boundaries, hard negatives, representation models, and training data—add an out-of-scope judgment module or change routing when necessary. Repeatedly fine-tuning thresholds may only cause validation set overfitting.
6. Designing Specific Clarification Questions
"What do you want to do" usually wastes information already obtained. Clarification should utilize what is already known, only asking about content that remains unclear and would affect subsequent processing.
| Known Situation | Less Effective Follow-up | More Specific Follow-up |
|---|---|---|
| User wants to book a room, missing dates | What service do you need | When do you want to check in, when to check out |
| Both order A and B could be referenced | Please add information | Do you want to cancel DEMO-A or DEMO-B |
| Hard to distinguish between consulting rules and executing cancellation | Please say again | Do you want to first learn about the cancellation fee, or proceed with cancellation |
| "Change it" has unclear object | What do you mean | Do you want to change hotel, dates, or room type |
| System lacks invoicing capability | Please select query, booking, or cancel | Current assistant doesn't support invoicing, you can contact the front desk; if needed, I can continue checking orders |
These specific options must come from current state or real candidates. If the model fabricates a non-existent order number, the follow-up question itself will cause confusion.
Clarification also needs to update state. For example, after the user selects order A, the system should bind the order slot to the current cancellation task and invalidate the old candidate question. When sending a follow-up question, also save the pending question to ensure the next turn can continue processing.
7. Setting Follow-up Question Limits
If the same gap isn't resolved after two consecutive attempts, continuing to ask with the same phrasing usually doesn't improve experience. The attachment provides a small strategy function, distinguishing missing slot, ambiguity, low support, and unsupported capability, demonstrating attempt limits:
Here the default limit is 2, only used to demonstrate strategy branches. Production systems should record attempts by gap or task, reset after problem resolution, and combine with user state to decide whether to offer a human handoff or end the task. This function doesn't implement a complete session counter and human system.
When the user explicitly abandons the current task, this decision should be prioritized; stop collecting parameters for that task. "Don't book" cancels the booking task in the conversation, not automatically converting to canceling a real order.
8. Evaluating Rejection Effects
Only reporting "accuracy among accepted samples" may hide large volumes of rejections. Need to simultaneously give coverage, error rate in acceptance, and the number of out-of-scope inputs mistakenly accepted.
Coverage = Number of samples that were accepted and selected a label / Total samples
Selective Error Rate = Error acceptance count / Accepted samples
When accepted samples are zero, selective error rate has no denominator—it should be recorded as undefined, not 0%. A system that rejects everything can have zero erroneous actions, but that doesn't mean it provides usable service.
Also should separately check the misrejection rate for known categories, misacceptance for each high-cost action, clarification success rate, and average follow-up question count. Single-round threshold experiments can't answer whether complete multi-turn tasks are completed—this part is left for subsequent evaluation articles.
9. Clarification Example for Order Cancellation
User says "cancel that," and the system has two orders. At this point, the cancellation target may be clear, but the object isn't unique—should follow up "Do you want to cancel DEMO-A or DEMO-B?" After the user selects A, the system also needs to verify the cancellation fee. If the user adds "if there's a fee, don't cancel," the condition should be written into the task, not continue with unconditional cancellation.
If querying the fee fails, the problem becomes service unavailability; if querying finds there is a fee, the problem becomes condition not met. Neither case should make the user explain their intent again. Each follow-up question only supplements currently missing information—the system then updates state based on the response and selects the subsequent flow.
The example also shows that clarification success rate can't be judged solely by whether the user responds. More useful judgment is: whether the response eliminates the original ambiguity, whether it correctly updates state, and whether it reduces erroneous actions.
Rejection means the system cannot currently reliably select a category. The system should decide whether to continue questioning, explain capability scope, or transfer to other services based on the reason.
Continue reading: Baseline Methods Tested, Label Design, Multi-turn State, Series Overview.