When designing an intent recognition system, you should first define the labels clearly, then choose the model. If both "cancel the charge" and "help me cancel" are labeled as "cancel order," the model grouping them together perfectly matches the training data, but the business may still execute the wrong action.
This article provides a reusable label definition methodology based on hotel booking and after-sales scenarios: how to choose granularity, how to differentiate intents from slots, when to use hierarchy or multi-labels, and how to maintain versions after business changes. The accompanying files contain definitions, positive examples, negative examples, allowed slots, and capability boundaries for 7 labels.
Download Label Specifications, Baseline Code, and Task Dependency Examples. Series overview at Intent Recognition in LLM Dialogue Systems.
1. Dividing Intents Based on Business Processes
When dividing intents, you can compare the processing flow, parameter requirements, permissions, and completion criteria of two requests. If the differences are significant, they should usually be divided into different labels; if only the city or date differs, they can be represented as slots under the same intent.
For example:
| Expression | What the System Needs to Do | Label |
|---|---|---|
| Will canceling the reservation charge a fee? | Query rules and explain | policy_query |
| Help me cancel order DEMO-A | Verify order and conditions, enter cancellation process | cancel_booking |
| How long for refund after cancellation? | Query refund timing or process | refund_query |
| Don't cancel, just check the order | Query existing order | order_query |
You cannot simply divide by keywords. The four sentences above all contain "cancel," but querying rules, executing cancellation, refund inquiry, and order query serve different functions.
You also shouldn't turn every API into a user intent. When a user wants to "check order status," the backend may first check the order, then payment, then fulfillment—these are implementation steps. If you directly use internal API names as labels, when backend interfaces change, the label system also needs modification.
2. Distinguishing Intents from Slots
"Book a hotel in Beijing" and "book a hotel in Shanghai" usually share book_room, with the city placed in a slot. If you create a label for each city, when a new city goes online, you must add categories, relabel data, and re-evaluate the model.
Slots are not just extracting entities from text, but also determining the entity's role in the task. "Depart from Beijing to Shanghai" has two cities, but only the task definition can determine which is the departure point and which is the destination. The hotel example requires clarifying which city to query listings for.
A label can be written as:
Here, "booking not yet submitted, change check-in date" is placed in booking. Only when modifying an existing order does it enter change_booking. Therefore, labels cannot be determined solely by the current sentence; you also need to know which stage the task is in.
The teaching_required_slots in the attachment is only used for teaching state machine calculations of missing parameters. Real bookings may also require specific products, guest names, prices, payment methods, etc.; the required fields in the example being complete only indicates it can proceed to subsequent business validation.
3. Choosing Appropriate Classification Granularity
Overly coarse labels mix different business branches together, still requiring a second recognition later. For example, "after-sales" may include cancellation, rescheduling, refund inquiry, and complaints; if downstream doesn't know which process to follow, this classification doesn't provide enough information.
When labels are too fine-grained, similar categories may be difficult to distinguish, and annotation consistency is harder to guarantee. "Inquire about breakfast time" and "Inquire about checkout time" if both handled by the same policy retrieval module can first share policy_query, with the topic as topic. If later the two types have different data, compliance requirements, or business metrics, consider splitting.
When judging whether to split a category, check:
- Whether there are stable, explainable user goal differences.
- Whether downstream can reduce ambiguity or choose different capabilities based on this.
- Whether annotators can consistently distinguish based on dialogue evidence.
- Whether each category has enough samples, and whether the maintenance benefit of new categories is clear.
"The model can't distinguish" doesn't automatically mean they should be merged—it could also be insufficient data. Conversely, "the model can distinguish" doesn't automatically mean it's necessary to split into business labels. First determine the processing requirements, then check model capabilities.
4. Writing Label Definitions
Beyond the label name, you also need to explain the applicable scope, negative examples, and conflict handling:
| Item | Content to Clarify |
|---|---|
| Stable ID | Identifier used in code and historical data, doesn't change with display name |
| Definition | The goal the user wants to achieve |
| Positive Examples | Various natural expressions and necessary context |
| Exclusions | Most easily confused similar requests |
| Allowed Slots | Parameters the task can accept |
| Stage Constraints | Whether existing order is required, whether continuing incomplete tasks is allowed |
| Capability Boundaries | Query only, allows changes, or system doesn't currently support |
| Uncertainty Handling | What information to add when evidence is insufficient, when to clarify |
| Version | When the definition takes effect, which old data needs review |
Positive and negative examples are best presented in pairs: "Cancel DEMO-A" vs. "Don't cancel DEMO-A first," "Check room price" vs. "Book at this price." Providing only typical positive examples doesn't explain how to distinguish similar requests.
Don't write facts that don't exist in the input or context into the answer. If data annotation must rely on annotator speculation, it should be changed to clarification examples or supplemented with truly available context.
5. Hotel Business Label Examples
| ID | Goal | Key Exclusions |
|---|---|---|
policy_query | Consult facilities, check-in, and cancellation rules | Actual operations, dedicated refund processes |
room_search | Query rooms, room types, and quotes not yet purchased | Submit reservation, query existing orders |
book_room | Establish new reservation and edit parameters before submission | Modify existing orders |
order_query | Query existing order information or status | Modify order, dedicated refund progress |
change_booking | Modify existing reservation | Create another new reservation |
cancel_booking | Request cancellation of existing reservation | Consult rules, denied cancellation |
refund_query | Consult refund process, amount, and arrival time | Direct payment or initiate refund |
Here, separating refund inquiry as a separate category is a business design choice in the example. Another system might classify it as a subclass of order query. As long as definitions are consistent and processing paths are reasonable, both designs can be valid; this table should not be treated as the standard answer for all hotel systems.
6. Hierarchical Classification and Multi-label Classification
Hierarchy is used to organize concepts from coarse to fine. For example, information/order, information/policy, transaction/cancellation. It can help with navigation, layered error analysis, and managing team responsibilities.
But whether to adopt two-stage classification needs actual testing. First misclassifying a request as information, then only selecting leaf labels within information, passes down the first error. Small-scale label catalogs can directly predict leaf IDs, with hierarchy only as a management structure.
Multi-label indicates multiple goals in the same input: "Check my order, and tell me breakfast time." It's not deeper hierarchy, but two tasks existing simultaneously.
There's also the case of same-type multi-task: "Book one room in Beijing, and another in Shanghai." If you save {book_room} as a set, the task count is lost. You should assign independent IDs to tasks, then save parameters separately. See Multi-label, Conditions, and Task Dependencies for specific structures.
7. Distinguishing Unknown Intent, Ambiguity, and Missing Parameters
"I want to book a room" has a clear goal but lacks city and date—continue filling slots. "Cancel that one" has referential ambiguity when there are two orders—ask for clarification. "Issue an invoice" in a system without invoice capability is an out-of-scope request.
These three cases should be represented using different state fields. If everything is classified as unknown, downstream programs cannot determine whether to supplement parameters, clarify the object, or explain that the feature isn't currently supported.
You also need to distinguish semantic scope from temporary availability. The system supports checking orders, but the order service is currently timing out—the intent is still order_query; this is a service failure and shouldn't be relabeled as out-of-scope. Conversely, if the system never had invoice capability, even if the model can recognize the invoice request, it should return that the feature is currently unsupported.
For "call me Big Hero in the future," a comprehensive assistant can pass it to the preference module; a hotel business recognizer can report the current scope doesn't support it. With different capability catalogs, reasonable labels will also differ.
8. Label Changes and Data Migration
Imagine invoice functionality goes live later. "Issue accommodation invoice" in old data was originally out-of-scope, but the new version should become invoice_request. The correct approach is to create a new label version, list affected samples, and switch routing after the new capability is available.
If splitting one coarse label into two fine labels, the old label often cannot automatically map to a unique new label. You need to review the original text and context—you can't just do string replacement. Although merging labels is easier to map, it may also change the denominator and business meaning of historical metrics.
You can adopt the following migration steps: first establish new definitions and difference explanations, sample and relabel; then replay fixed data with both old and new definitions to check routing changes; subsequently update labels, models, Prompts, data, and interface versions; retain old versions needed for rollback. Which label version historical reports use should also be recorded.
9. Unifying Label Definitions in Code
The attachment batch2/data/taxonomy.json saves the complete catalog, with leaf IDs consistent with the first batch of core.py. You can check in the project root:
Using the same catalog for label specifications, model input, and service validation avoids different modules using inconsistent definitions. For example, when adding a new slot, after updating the specification, you should synchronously generate Prompts and validation rules, rather than maintaining three separate strings.
Label definitions should be used simultaneously for data annotation, model recognition, and business processing. Organize easily confused requests into positive and negative examples, then supplement corpora and compare models based on this to determine whether improvements are effective.
Continue reading: Complex Intent Corpus, Three Basic Recognition Methods Tested, Unknown Intent and Clarification Strategies.