A classifier that returns correct labels in a script can still encounter a different class of problems once integrated as a service: requests timeout and get resubmitted, two responses overwrite the same session state, the user has already changed the date, but the delayed result still executes with the old parameters.
Addressing these issues requires attention to the entire process from receiving requests to saving results. This article covers request identification, state versioning, idempotency, timeouts, caching, and error sample handling, along with a local HTTP service for running and testing. Download Service and Test Cases.
Request deduplication, state checking, business logic, and atomic commit flow
One: Designing Request Interfaces
A stateful recognition request needs, in addition to the text, to know which session it belongs to, whether it's a retry, and which state version it was based on.
{
"session_id": "hotel-demo",
"request_id": "turn-002",
"expected_version": 1,
"text": "Beijing 2026-10-01 2026-10-03",
"budget_ms": 2000
}
session_id locates the session, request_id identifies one logical request, and expected_version represents the state version the caller observed. The three fields are not interchangeable: a single session can have many requests, the same request may be sent multiple times, and many different requests can also be based on the same old version.
Real services also need to obtain user identity from the authentication context, binding the session and business objects to that identity. User-provided text in the request body cannot replace authentication. The attached demo uses a fixed demo identity, and the interface only listens on the loopback address, making it easy for readers to verify the flow.
Two: Request Idempotency and State Versioning
The first request succeeded, but the response was lost in the network. When the client retries, it should reuse the original request_id. When the service finds this logical request has already completed, it directly returns the saved result, rather than updating the state or canceling the order again.
If the same request ID brings different text, it's not a normal retry. You can save fingerprints of key request fields and return a conflict when different content is detected, preventing old results from being incorrectly used for new requirements.
State versioning handles another scenario: two different requests are both based on version 3, one modifies the date, the other confirms the booking. If the modification is submitted first to version 4, the confirmation cannot continue to overwrite version 4. Conditional updates should explicitly require that the current version is still 3, otherwise return a conflict and let the client read the state again.
| Scenario | Expected Behavior |
|---|---|
| Same request ID, same content, result exists | Return original result |
| Same request ID, different content | Idempotency conflict |
| New request ID, state version too old | State conflict |
| New request ID, version matches | Continue processing and commit new version |
The check order also matters. The old version carried by normal retries may have already been advanced by the first successful request. Therefore, you should first check completed request records, then determine whether the new request is based on stale state.
Three: Ensuring Data Consistency with Transactions
If the service first updates the state, then saves the deduplication record, and crashes in between, retries might execute again. The attachment puts session state, request results, audit events, and simulated order operations in a single SQLite transaction: either save everything or roll back everything.
SQLite provides transaction and commit/rollback mechanisms, making it suitable for demonstrating these state consistency relationships. Python sqlite3 documentation
Real order systems are usually in another service and cannot directly join the local database transaction. In that case, at minimum pass the logical operation ID to the tool so the tool itself supports idempotency; if the call times out, first query the result by operation ID, then decide whether to retry. When necessary, use outbox patterns, task state machines, and result reconciliation to handle cross-system consistency.
It's particularly important to distinguish between "call returned failure" and "business operation did not occur." These can differ when the connection disconnects after commit. Immediately换一个新请求 ID 再调用，会绕过去重机制，产生重复操作。
Four: Setting Timeout and Retry Strategies
A request has a total budget of two seconds; you can't let retrieval, model, and tool each wait two seconds. You should save the absolute deadline, calculate remaining budget at each step, then decide whether to continue, degrade, or return a clear service result.
Total time can be split into: queuing, reading state, recognition, retrieval or tool, commit, and response. Only by splitting can you know whether optimizing the model improves user experience. If half the time is spent queuing, compressing model inference alone has limited benefit.
Retries consume remaining budget and increase system load. Typically suitable for retry are transient network or service failures; parameter errors, permission denials, and state conflicts often require changing the request. Exponential backoff and random jitter can prevent large numbers of clients from synchronized retries, but you still need to set an upper limit on attempts and ensure the operation is retry-safe. AWS timeout, retry, and jitter documentation
The attachment uses monotonic clocks to check deadlines at the processing node and rolls back when the budget is exceeded. This is cooperative budget checking: ongoing blocking calls still need client timeout; CPU inference for forced cancellation requires interruptible execution units or process isolation. The budget field in requests needs to work with these execution mechanisms to actually limit execution time.
Five: Transaction Locks and Concurrency Control
The sample code puts the entire local request processing in one transaction for easy demonstration of how state and simulated orders commit together. When model inference is slow, the transaction holds the lock continuously, so high-concurrency services need to adjust transaction scope.
Real services can first read a state snapshot, complete expensive recognition outside the transaction, then do a short transaction commit with expected_version. You must re-verify the version at commit time; if the state has changed during recognition, discard results based on the old snapshot or recalculate.
Write tools also need finer-grained task states, such as proposed, confirmed, executing, succeeded, failed, and unknown. unknown means the result is pending reconciliation and cannot be treated as a clear failure. Separating these states lets recovery logic know whether to re-call, query the result, or wait for user handling.
Python standard library HTTP servers can be used for local demos; production deployments should choose servers suitable for authentication, concurrency, connection management, and observability, and set up reverse proxies and resource limits. http.server documentation
Six: Designing Cache Keys and Invalidation Strategies
Two users both input "1" may select different orders; the same user entering "change to tomorrow" at different times may modify different tasks. Caching recognition results by text alone will carry one context's interpretation into another.
Cache keys should cover all inputs that affect results: current text, necessary history or state summary, label version, model and prompt version, language, and appropriate user or tenant isolation information. Different summaries may lead to different judgments, so summary strategy version should also be included in the version combination.
Recognition cache and business result cache should also be separate. Caching a policy classification candidate has different expiration and invalidation conditions than caching an order's current state. Even more importantly, don't treat "once confirmed cancellation" as an infinitely reusable confirmation cache.
If identity boundaries or state dependencies are difficult to define clearly, not caching is usually easier to maintain than using wrong cache. First measure cost hotspots, then decide which layer to cache.
Seven: Service Monitoring and Business Metrics
Looking only at requests per second and error codes makes it hard to discover when the model routes a consultation as an operation. You can divide observations into three perspectives:
| Perspective | Focus | Use Case |
|---|---|---|
| Service | Latency distribution, timeouts, queuing, dependency failures | Locating runtime bottlenecks |
| Decision | Label distribution, rejection, clarification, model upgrade ratio | Observing input and strategy changes |
| Task | Completion, abandonment, human takeover, duplicate and erroneous operations | Judging actual user outcomes |
Record label, model, prompt, routing, and state version for each request to correlate changes with specific releases. Keep enough structured fields in logs for diagnosis; handle raw conversations according to clear permission, desensitization, and retention policies; don't default to writing all chat and business parameters to public logs.
The local example's audit table only saves session and request identifiers, version, routing, and latency. Production environments can additionally connect trace ID, tool operation ID, and user feedback to form cross-service request chains.
Eight: Error Sample Review and Version Release
When the user corrects "I didn't want to cancel," it's a valuable error signal, but you still need to review the context: did the model misunderstand, was the interface unclear, or did the user change their mind?
You can first collect signals, then group by error type, review labels and expected actions, and add to training or regression data. Automatically labeling all online failures as one category mixes tool failures, product issues, and user changed minds into the recognition training.
When releasing, save a set of rollbackable versions: labels, data, prompts, models, thresholds, and routing. Shadow mode only compares old and new outputs without executing new方案的 business writes; in canary releases, gradually open real traffic while continuously monitoring high-cost errors. After updating the model, you also need to verify the complete business flow.
Nine: Starting and Testing HTTP Service
python batch3/service.py --db /tmp/intent-demo.sqlite3 --port 8765
In another terminal, make a request:
curl http://127.0.0.1:8765/turn \
-H 'Content-Type: application/json' \
-d '{"session_id":"demo","request_id":"r1","expected_version":0,"text":"Cancel order DEMO-A"}'
The system first returns confirm. Only by using a new request ID, the returned version, and the text "confirm" will the simulated operation complete. Sending the exact same confirm request repeatedly returns the first saved result.
You can also run the automated checks directly:
python batch3/tests.py
python batch3/http_smoke.py
This real loopback HTTP check shows: propose and confirm success return 200, original request replay returns the same result, old version new request returns 409, same ID with different content returns 409, extra fields return 400. Fault tests also verify timeout rollback, order version changes, fee condition changes, and state recovery after restart.
The next article Complete Assistant Project connects labels, recognition, state, routing, and these service constraints, providing a path from installation to complete conversation.