Even when a recognizer returns the correct label in a script, integrating it into a service can still surface a different class of problems: timed-out requests may be resubmitted, two responses may overwrite the same session state, and a late result may still execute with stale parameters even after the user has changed the date.
Solving these problems means focusing on the entire path from receiving a request to saving the result. This article introduces request identifiers, state versions, idempotency, timeouts, caching, and error-sample handling, and includes a local HTTP service you can run and test. Download the service and failure cases.
Flow of request deduplication, state checks, business decisions, and atomic commit
I. Designing the Request Interface
A stateful recognition request needs to specify not just the text, but also which session it belongs to, whether it is a retry, and which state version it is based on.
session_id identifies the session, request_id identifies one logical request, and expected_version represents the state version the caller has seen. None of the three fields can substitute for another: a session can contain many requests, a single request may be sent multiple times, and several distinct requests may all be based on the same old version.
A production service also needs to obtain the user's identity from the authentication context and bind the session and business objects to that identity. A user's claim in the request body cannot replace authentication. The attached package always uses a fixed demo identity, and the service listens only on the loopback address so readers can verify the flow.
II. Request Idempotency and State Versions
The first request succeeds, but its response is lost in transit. When the client retries, it should reuse the original request_id. If the server finds that the logical request is already complete, it should return the saved result directly rather than update the state or cancel the order again.
If the same request ID arrives with different text, it is not a normal retry. The server can store a fingerprint of the request's key fields and return a conflict when the incoming content differs, preventing an old result from being mistakenly applied to a new intent.
State versions handle another case: two different requests are both based on version 3, with one changing the date and the other confirming the booking. If the change commits as version 4 first, the confirmation must not go on to overwrite version 4. A conditional update should explicitly require the current version to still be 3; otherwise, it should return a conflict so the client can reload the state.
| Case | Appropriate behavior |
|---|---|
| Same request ID, same content, result exists | Return the original result |
| Same request ID, different content | Idempotency conflict |
| New request ID, stale state version | State conflict |
| New request ID, matching version | Continue processing and commit the new version |
Check order matters as well. The stale version carried by a normal retry may already have been superseded by the first successful request. You should therefore check the completed-request record before deciding whether a new request is based on stale state.
III. Using Transactions to Ensure Data Consistency
If a service updates state first and saves the deduplication record second, a crash in between can cause a retry to execute again. The attached package places the session state, request result, audit events, and simulated order operation in a single SQLite transaction: either everything is saved, or everything is rolled back.
SQLite supports transactions with commit and rollback mechanisms, making it suitable for demonstrating how these pieces of state stay consistent. Python sqlite3 documentation
A real order system usually runs as a separate service and cannot be added directly to the local database transaction. At a minimum, pass the logical operation ID to the tool and have the tool support idempotency. If the call times out, query the result by operation ID before deciding whether to retry. When necessary, use an outbox, a task state machine, and result reconciliation to maintain consistency across systems.
It is especially important to distinguish “the call returned a failure” from “the business operation did not happen.” If the connection drops after the commit, the two may differ. Calling again immediately with a new request ID bypasses deduplication and produces a duplicate operation.
IV. Setting Timeouts and Retry Policies
A request has a total budget of two seconds. Do not give retrieval, the model, or each tool call its own two-second timeout. Save an absolute deadline, calculate the remaining budget at each step, and then decide whether to continue, degrade gracefully, or return an explicit service result.
You can break total elapsed time down into queueing, state reads, recognition, retrieval or tool use, and commit and response. Only then can you tell whether optimizing the model will improve the user experience. If half the time is spent queueing, reducing model inference time alone will have limited impact.
Retries consume the remaining budget and increase system load. Transient network or service failures are usually suitable for retries; invalid parameters, permission denials, and state conflicts generally require changing the request. Exponential backoff and jitter can keep many clients from retrying in sync, but you still need to cap the number of attempts and make sure the operation is safe to retry. AWS guide to timeouts, retries, and jitter
The attached package uses a monotonic clock to check the deadline at each processing step and rolls back if the budget is exceeded. This is cooperative budget checking: blocking network calls still need client-side timeouts, and forcibly canceling CPU inference requires interruptible execution units or process isolation. The request's budget field must work with these execution mechanisms to cap the actual elapsed time.
V. Transaction Locks and Concurrency Control
The example code places the local processing of the entire request in one transaction, making it easy to show how the state and simulated order are committed together. When model inference is slow, the transaction continues to hold a lock, so a high-concurrency service needs to adjust the transaction scope.
A production service can first read a state snapshot, complete the expensive recognition work outside the transaction, and then use expected_version to commit in a short transaction. The version must be revalidated at commit time. If the state changed during recognition, discard the result based on the old snapshot or recompute it.
Tools that perform writes also need more granular task states, such as proposed, confirmed, executing, succeeded, failed, unknown. unknown means that the result is still awaiting reconciliation and must not be treated as a definite failure. Keeping these states separate lets the recovery logic know whether to call again, query the result, or wait for user intervention.
Python's standard library HTTP server is suitable for a local demo. A production deployment should choose a server suited to authentication, concurrency, connection management, and observability, with a reverse proxy and resource limits configured. http.server documentation
VI. Designing Cache Keys and Invalidation Strategies
Two users may both enter “1” but select different orders. The same user may enter “change it to tomorrow” at different points and modify different tasks. Caching recognition results by text alone carries an interpretation from one context into another.
The cache key should cover every input that can affect the result: the current text, any necessary history or state summary, the label version, the model and prompt versions, the language, and appropriate user- or tenant-isolation information. Different summaries can lead to different decisions, so the cache's version tuple should also include the summarization-strategy version.
Recognition caches and business-result caches should also be kept separate. Caching a candidate policy classification and caching an order's current state have different validity periods and invalidation conditions. And you should never treat “cancellation was previously confirmed” as a confirmation cache that can be reused indefinitely.
If identity boundaries or state dependencies are difficult to define, skipping the cache is usually easier to maintain than using the wrong cache. Measure the cost hotspots first, then decide which layer to cache.
VII. Service Monitoring and Business Metrics
Looking only at requests per second and error codes makes it hard to spot when the model routes an inquiry to an action. You can organize observability around three perspectives:
| Perspective | Focus | Purpose |
|---|---|---|
| Service | Latency distribution, timeouts, queueing, dependency failures | Identify runtime bottlenecks |
| Decision | Label distribution, recognition rejection, clarification, and the proportion of cases escalated to a larger model | Observe changes in inputs and policies |
| Task | Completion, abandonment, human intervention, duplicate and erroneous operations | Determine actual user outcomes |
For every request, record the label, model, prompt, route, and state version so that changes can be tied to a specific release. Keep enough structured fields in the logs to support diagnosis, and handle raw conversations under explicit access, redaction, and retention policies. Do not default to writing all chat and business parameters to public logs.
The local example's audit table stores only the session and request identifiers, version, route, and elapsed time. In production, you can also correlate the trace ID, tool operation ID, and user feedback to form a request chain across services.
VIII. Reviewing Error Cases and Releasing New Versions
When a user corrects the system with “I didn't mean to cancel,” that is a valuable error signal. You still need to review the context: Did the model misunderstand, was the information unclear in the interface, or did the user change their mind temporarily?
Start by collecting signals, then group them by error type, review the labels and intended actions, and add the cases to training or regression data. Automatically assigning every production failure to one category would mix tool failures, product issues, and changes in user intent into the training data for intent recognition.
At release time, keep a rollback-ready set of versions for the labels, data, prompt, model, thresholds, and routing. Shadow mode compares only the old and new outputs and does not perform any business writes for the new approach. During a canary release, gradually expose live traffic and continue monitoring costly errors. After updating the model, you also need to validate the complete business flow.
IX. Starting and Testing the HTTP Service
python batch3/service.py --db /tmp/intent-demo.sqlite3 --port 8765
Send a request from another terminal:
The system first returns confirm. To complete the simulated operation, continue with a new request ID, the version returned in the response, and the text “Confirm.” Sending the exact same confirmation request again returns the result saved the first time.
You can also run the automated checks directly:
python batch3/tests.py
python batch3/http_smoke.py
This run of the actual loopback HTTP checks produced the following results: the action proposal and confirmation succeeded with 200 responses; replaying the original request returned the same result; a new request with an old version returned 409; the same ID with different content returned 409; and an extra field returned 400. The failure tests also covered rollback after a timeout, order version changes, changes in fee conditions, and state recovery after a restart.
The next post, The Complete Assistant Project, connects labels, recognition, state, routing, and these service constraints, providing a walkthrough from installation through a complete conversation.