When implementing intent recognition, you can start by comparing three fundamental approaches: keyword rules, TF-IDF with linear classifiers, and sentence vector nearest neighbor search. They rely on explicit conditions, business annotations, and pre-trained semantic representations respectively, serving as baselines for subsequent optimization.
This article runs three recognizers on the same set of Chinese hotel teaching data, revealing data splits, implementation parameters, threshold selection, and prediction-by-prediction results. Through comparison, we'll see why the three methods judge differently and how to choose the next optimization step based on errors.
Download code, data, and complete experimental results. Label definitions are in Intent Label Design, and complex multi-turn examples are in 48 Corpus Special.
1. Task Definition and Data Preparation
This time we only recognize the business category of single-turn sentences, without extracting slots, restoring history, or executing tools. We use 7 labels: policy inquiry, room check, booking, order query, modify booking, cancel booking, and refund inquiry.
To compare three methods, we prepare three datasets for different purposes:
| File | Composition | Purpose |
|---|---|---|
| train.jsonl | 42 items, 6 per 7 categories | Fit TF-IDF and classifier, build vector example library |
| validation.jsonl | 21 known categories + 7 out-of-scope | Select score threshold and candidate score difference |
| probe.jsonl | 14 known categories + 7 out-of-scope | Demonstration check after parameter fixation |
This teaching data is used below to explain method differences. Complete data and runtime environment are in the download package README.
The training set is used for learning, the validation set for threshold selection, and the probe set for observing fixed方案的行为. Separating these three uses is the only way to know where improvements come from.
2. Unified Recognizer Interface
All three recognizers implement score(texts), returning a mapping from labels to scores for each input sentence. Their score meanings differ: rules output hit or miss, TF-IDF classifier outputs probability, and sentence vectors output the highest cosine similarity to examples in that category.
These methods use the same return structure, but the meaning and scale of scores differ. You cannot assume two schemes have the same reliability just because both return 0.7. Whether to select a label is determined by each method's validation data and decision strategy.
The candidate interface also does not directly replace the complete structured output of the first batch. Classifiers lacking slots and context should not return field values that haven't been extracted or inferred. Subsequent routing can first use candidates, then call parameter extraction or dialog state modules.
3. Keyword and Regex Rules
Rule implementation is simple: each category has several regex patterns, returning 1 if any match, otherwise 0. For example, "查.*订单" is for order inquiry, and "退款" is for refund inquiry.
Rules select classification branches through text patterns. They are easy to explain and execute, but the patterns themselves usually don't know which action the negation modifies or which target the condition constrains.
"不要取消，只查订单" may match two categories simultaneously. This article chooses clarification when scores tie for highest, rather than letting dictionary order decide the final action. When nothing matches, it also doesn't default to the first label.
Production rules can be finer, but you must monitor rule mutual masking, order changes, and the impact of new exceptions. After adding rules, testing is still needed to confirm whether coverage improved and whether new conflicts were introduced.
4. TF-IDF and Logistic Regression Classification
Chinese words typically have no spaces between them; this example directly uses character n-grams for feature extraction. This example uses continuous fragments of 2 to 4 characters, fitting vocabulary, IDF, and logistic regression classifier on training data:
Character fragments can capture clues like "退款" (refund), "到账" (arrived), "订单" (order), but may also learn accidental features like templates or order number prefixes. Real experiments should check performance after anonymization and entity replacement, and verify whether the model relies on irrelevant features like order number formats.
Here, TF-IDF converts text to sparse features, and logistic regression learns the influence of each feature on different labels. TfidfVectorizer documentation
TF-IDF can be understood as two quantities. TF describes a fragment's occurrence in the current text, and IDF reduces the weight of fragments that appear everywhere. If "酒店" (hotel) appears in almost every category, it provides little distinguishing information; "到账" (arrived) concentrated in refund category is often more helpful.
Logistic regression then weights these features: for each category calculate z_k = w_k · x + b_k, and the multi-class model converts these scores through softmax into a distribution. During training, weights are corrected according to annotations so the correct category gets higher scores. Its difference from hand-written rules is that the positive/negative effects of features on categories are learned from data, rather than entirely prescribed by developers.
Character n-grams also explain its weaknesses. "退回款项" (return money) and "钱什么时候到账" (when will money arrive) may lack the same fragments; when synonymous expressions have wide spans, sparse features can hardly establish connections on their own. Extending n-grams can preserve local word order but increases sparsity; too short easily loses phrase meaning.
The model's predict_proba output also cannot automatically solve out-of-scope input. The probability distribution over known categories still sums to 1, and a completely unrelated request will also have a maximum category. Additional rejection experiments are needed, rather than directly routing by maximum value.
5. Sentence Vector and Nearest Neighbor Classification
This example loads the multilingual sentence vector model sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, runs on CPU, and fixes the model hub revision. Both training example sentences and queries are vector normalized, then cosine similarity is computed via dot product.
The model outputs 384-dimensional representations, turning each sentence into a fixed-length vector. Model card
Sentence Transformers provides sentence encoding and similarity interfaces; the specific comparison method and normalization configuration should remain consistent. Similarity documentation
After normalization, vector length is 1, so dot product equals cosine similarity. The model maps semantically similar expressions to nearby directions, so "钱何时回来" (when will money come back) and "退款多久到账" (how long for refund to arrive) may be close even with little literal overlap. The key capability here comes from encoder pre-training, not from training on the current 42 example sentences.
But semantic similarity doesn't mean identical intent. "取消订单" (cancel order) and "取消订单会收费吗" (will canceling order charge fee) have very close topics but should go into two different branches: operation and consultation. This explains why vector methods need boundary counter-examples, and why general semantic similarity cannot directly equal business decision-making.
Example-level retrieval and category centers each have costs. Category centers average same-category vectors for comparison, making storage and retrieval simpler, but may lose features of different expressions within same-category examples; example-level retrieval preserves diversity but is more easily affected by single mislabeled or accidentally similar examples. This article actually runs "highest example similarity per category"; the category center scheme did not participate in this comparison.
Model input length is also a limitation. Directly concatenating long history into the sentence vector encoder may truncate critical context; this experiment only processes short sentences and does not infer multi-turn capability from this.
6. Selecting Rejection Thresholds
Each method uses the validation set to search for the highest score threshold and top-two score difference. The teaching objective function is "5× wrong acceptance + 1× known input not selected", prioritizing reduction of wrong acceptance while accepting certain rejection costs.
Rule hits, classification probability, and cosine similarity using the same candidate grid is only for demonstration convenience; ultimately each selects parameters separately. Thresholds are not universal constants; after labels, examples, or models change, they need rechecking.
For more complete explanation, see Unknown Intent, Rejection, and Clarification. Rejection means current evidence is insufficient to choose, not that it's already confirmed the user is outside business scope.
7. Classification Results and Error Analysis
| Method | Known Input Raw Top-1 | Accept Count / 21 | Accept Error Count | Out-of-Scope False Acceptance / 7 |
|---|---|---|---|---|
| Keyword Rules | 8/14 | 10/21 | 3 | 1/7 |
| TF-IDF + Logistic Regression | 9/14 | 6/21 | 1 | 1/7 |
| Sentence Vector Nearest Neighbor | 10/14 | 4/21 | 0 | 0/7 |
Raw Top-1 only looks at 14 known inputs, does not include out-of-scope samples, and does not use rejection strategy. It helps separate "candidate ranking ability" from "whether to accept decision". When rules have zero hits, raw ranking breaks ties by label order; this result is only used as a classification reference without rejection strategy, and actual strategy will not accept zero hits or tied highest scores.
Post-threshold accept count uses all 21 probe inputs as denominator; accepted errors include known category misclassification and out-of-scope false acceptance. Therefore, observe both ranking and rejection: the former answers "which category is most similar", the latter answers "is evidence sufficient now to make a decision".
The following three sentences correspond to recognition errors in negative expressions, existing order modification, and out-of-scope requests respectively:
| Input | Correct Category | Rule Decision | TF-IDF Decision | Sentence Vector Decision |
|---|---|---|---|---|
| 别取消了，把 DEMO-Z 的订单详情给我看看 | Check Order | Cancel Booking | Need to distinguish candidates | Rejection |
| DEMO-Z 这张订单换个房型 | Modify Booking | Check Room | Need to distinguish candidates | Rejection |
| 明天的足球比赛几点开始 | Out of Scope | Policy Inquiry | Policy Inquiry | Rejection |
The first sentence requires understanding that "别" negates cancellation; the second sentence's "换房型" (change room type) occurs on an existing order and should enter the modification process; the third sentence, although containing "几点" (what time), talks about football matches. They show: keywords, local characters, and semantic similarity each provide part of the evidence; business goals also depend on negation scope, task stage, and capability catalog.
Besides total scores, specific error cases should also be analyzed. If cancel policy is recognized as cancel action, check label positive/negative examples and semantic boundaries; if invoicing is recognized as order query, check out-of-scope samples; if recognition fails after rephrasing, check whether text features and training samples cover such expressions.
8. Inference Latency Comparison
| Method | Single Inference Median | This Sample P95 |
|---|---|---|
| Keyword Rules | 0.005 ms | 0.005 ms |
| TF-IDF + Logistic Regression | 0.100 ms | 0.203 ms |
| Sentence Vector Nearest Neighbor | 4.542 ms | 4.936 ms |
Timing uses per-sentence inference after CPU warm-up, excluding model loading; complete environment is in results file. Rules only do pattern matching, TF-IDF requires sparse features and linear operations, and sentence vectors also need neural network encoding; these compute paths explain the latency differences.
Online cost also depends on calling method. Vectors for fixed examples can be pre-computed; multiple inputs can be batch encoded; high-frequency, certain rule entries can reduce model calls. Before optimization, first distinguish initialization, single inference, and queuing time to find the real bottleneck.
9. Choosing Methods Based on Business Requirements
You can arrange experiments according to task constraints, rather than first deciding you must use a certain type of model.
For fixed menus and few clear commands, start with rules. When labels are stable and annotations already exist, TF-IDF with linear classifier can provide a lower-cost comparison starting point. When hoping to quickly cover synonymous expressions by adding example sentences, try vector nearest neighbor, but specially check negation and out-of-scope inputs. When complex context explanation is needed, then compare large models or suitable specialized models.
For further understanding of fine-grained intent classification, you can read BANKING77 related research, learning how to represent example sentences with bi-encoders and distinguish similar categories. BANKING77 Paper
Fine-tuning small models, distillation, and online layered routing require more data and experiments; this article first establishes three runnable baselines. Before adopting more complex methods, first clarify which discovered problems need solving, then verify effects through experiments.
10. Running Experiment Code
Create an isolated environment in the download package root directory:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r batch2/requirements.txt
python batch2/run_experiment.py
python batch2/tests.py
The first run of the sentence vector scheme requires downloading weights from the model hub. Model ID and revision are stored in batch2/model.json, library version and original results are in batch2/results/experiment.json. The download package only contains code, data, configuration, and experimental results; it does not redistribute model weights.
To run only rules and TF-IDF, use python batch2/run_experiment.py --without-embeddings; the current complete requirements still install experiment dependencies; readers can build a minimal environment after confirming dependencies. Offline strategy and task graph tests use only the standard library.
Keep the baseline's label definitions, training data, scores, rejection results, and error cases for comparing subsequent changes. After supplementing data or changing models, you can check which errors were fixed and which categories' performance declined.
Continue reading: Label Design, Large Model Structured Recognition, Unknown Intent and Clarification.