From RAG to Agent: Engineering Design for Production-Grade AI Systems

A knowledge-base Q&A demo usually only needs three steps wired together: splitting documents, retrieving chunks, and calling the model. Once it goes live, the questions become much more concrete: If the answer is clearly in the document, why does the model still get it wrong? Why does the task disappear when the user refreshes the page? After a tool request times out, is it safe to retry? As traffic grows, which runs out of headroom first—the connections, the database, or the model quota?

These questions span information retrieval, distributed systems, and model applications. Simply memorizing terms such as RAG, Rerank, and Checkpoint isn't enough to explain how they relate.

This article uses an enterprise knowledge-base assistant as an example: users upload documents, ask follow-up questions about them, and, when necessary, have the system query data, call tools, and generate reports. Following this task, we'll explain how retrieval supplies evidence, how context organizes that evidence, how an Agent advances execution, and how the system keeps working after failures.

The candidate counts, budgets, and capacity figures in this article are illustrative examples for explaining the design. They need to be evaluated and load-tested against real data; don't treat them as production configuration as-is.

1. Start by distinguishing the knowledge flow, execution flow, and state

A system can first be divided according to the responsibilities below. The modules in the diagram are logical boundaries; a small project can perfectly well deploy them all in one service.

Two facts are easy to conflate here.

First, RAG and Agent can be combined, but neither requires the other. A fixed “retrieve, then answer” flow can be implemented as a Workflow; an open-ended investigation can be handled by an Agent deciding what to look up next, with retrieval as just one of its tools.

Second, even if a single model call depends only on the messages passed to it, the application as a whole is still stateful. The system must manage who the user is, where the current task has reached, which tools have succeeded, and how much budget remains. The model's context is only part of it.

So start the design by answering three questions: Where does the knowledge come from? How does the task advance? Where are execution facts stored? Component selection comes after those questions.

2. Knowledge ingestion: parsing quality sets the ceiling for retrieval

The first step in ingesting a document is parsing. Converting a PDF directly into one long string can lose heading hierarchy, table relationships, figure captions, and page numbers. Even if the Embedding is strong afterward, it can't recover facts that were lost.

A better intermediate representation is “document—section—content block.” A content block can be a paragraph, table, image, or code snippet, while preserving its source location. A split chunk might contain:

The body text is used for retrieval and answering; the document version handles conflicts between old and new content; the page number and location support citations; and tenant and permission information enforce access control. These associations should be established at ingestion time.

Cleaning also needs restraint. Repeated headers and navigation bars are usually noise, but units in tables, negations in contracts, and exceptions in footnotes can directly affect the answer. The goal of cleaning is to remove interference while preserving meaning.

The same body of knowledge can have both a full-text index and a vector index. When updating a document, you also need to decide when each index becomes visible: if the full-text index is already at v3 while the vector index is still at v2, a single query could mix contradictory versions.

One viable design is to assign a version to each index build, switch the active, queryable version only after completion, and have every candidate retain its document version. Deletions and permission revocations must propagate to indexes and caches in sync; changing only the source file isn't enough.

3. The key to chunks is retrieval granularity and semantic completeness

Fixed-length splitting is easy to implement, but it doesn't understand whether a sentence is complete or how rules relate to their exceptions.

For example, a refund rule has two conditions: the purchase is within the deadline and the product hasn't been activated. If the split boundary falls exactly between them and only one block is retrieved, the system may reach the wrong conclusion.

Different splitting methods serve different purposes:

Overlap can mitigate boundary problems, but it increases storage, duplicate retrieval, and context overhead. It also can't guarantee that clauses far apart will be retrieved together.

For long, clearly structured documents, you can use Parent–Child Retrieval: use smaller chunks for retrieval, then add the containing subsection or necessary adjacent chunks after a hit. This preserves targeting precision while retaining the context needed to answer. The added scope must still be constrained by the token budget; don't dump an entire chapter on the model every time there's a hit.

When a document runs to hundreds of thousands of words, you can also build a hierarchical index of document summaries, section summaries, and original chunks. Hierarchical routing can narrow the search space, but if the first step selects the wrong section, later steps have no chance of finding the answer. It's therefore best to retain a fallback path for cross-section or global retrieval.

The granularity used for retrieval can be finer than the granularity provided when generating an answer. This distinction is more useful than fixing one Chunk size.

4. Hybrid retrieval: let exact matching and semantic matching complement each other

When a user asks “What should I do if my computer freezes?” while the document says “How to handle an unresponsive computer,” vector retrieval may recognize the semantic connection. When a user enters ORA-00942, an order number, or an API name, term matching is often more direct.

The main factors in BM25 are term frequency, inverse document frequency, and document-length normalization. The term-frequency contribution gradually saturates; rarer terms are usually more discriminative; and length normalization prevents long documents from winning solely on term frequency. The actual results of Lucene BM25Similarity documentation also depend on the tokenizer, field settings, and indexing granularity, so “we use BM25” doesn't mean exact matching will necessarily succeed.

Vector retrieval maps the Query and content to vectors, then finds candidates using a selected distance or similarity metric. Large-scale retrieval typically uses approximate nearest-neighbor algorithms, requiring trade-offs among recall, latency, and indexing cost. Query vectors and document vectors must follow the same encoding conventions of the retrieval model; when upgrading the model, plan for re-embedding and an index switch.

When combining the two result sets, don't simply add the raw scores. A BM25 score of 12.8 and a vector similarity of 0.86 don't share a natural scale. You can calibrate scores against evaluation data, or start with rank-based Reciprocal Rank Fusion:

RRF(d) = Σ 1 / (k + rank_i(d))

Here, the sum covers only the result lists in which document d appears, ranks start at 1, and k is the smoothing constant. For example, with k = 60, a result ranked first and second in the two lists scores 1/61 + 1/62. RRF avoids directly aligning different raw score scales, but it discards score-gap information and isn't guaranteed to be optimal across all datasets. Elasticsearch RRF documentation

The full flow can be:

Query rewriting isn't required every time. When a user asks “Why is it slower now?”, the system needs to recover the object from the conversation history. When the user has already supplied a complete error code, additional rewriting may add latency or even alter a key term. Preserve the original question and record both the pre- and post-rewrite versions.

Permission filtering must be generated server-side from a trusted identity and pushed down into both retrieval paths as far as possible. Check access again before handing content to an external reranking service or model. Handing unauthorized content to the model and hiding the citation only at the end is too late.

5. Rerank improves ranking precision—and can also become a source of latency

Common Bi-Encoders encode the Query and document separately, so document vectors can be computed ahead of time. A Cross-Encoder reads the Query and candidate content together and scores relevance through their interaction, making it suitable for reranking a small set of candidates. It must process each query-candidate pair, so it isn't suitable for exhaustive use over a large-scale corpus. Sentence Transformers: Retrieve & Re-Rank

This is why the pattern is “retrieve first, then rerank.” A reranker can only improve candidates that have already been retrieved; it cannot recover evidence missed earlier.

If reranking is too slow, start by examining the candidate count, candidate length, and queue wait time. You can remove duplicate candidates, adjust the count to the difficulty of the question, cap clearly irrelevant long passages, and then evaluate batching, a smaller model, quantization, or inference-backend optimizations.

Fewer candidates isn't always better. Pruning too early can lose evidence, while oversized batches may increase throughput but make low-traffic requests wait longer. Optimization should measure both retrieval quality and tail latency.

The cache key should at least distinguish the Query, document content version, and reranking model version. Even when a request is served from cache, the current permission check must still run.

If reranking times out, you can fall back to the ordering from hybrid retrieval, but that's a quality degradation. Routine informational queries may accept it; workflows with strict evidence requirements should instead fail explicitly or ask for more information. The fallback policy needs to be recorded in the Trace so that quality changes can be explained later.

6. After retrieval hits, ensure the evidence actually makes it into the answer

“The right evidence was retrieved, but the answer is wrong” is not a contradiction. The correct chunk may drop out after reranking, be truncated while the Prompt is assembled, or enter the context alongside an outdated document and then be misinterpreted by the model.

When debugging, you need to see the following sets:

检索得到的候选
    ↓
重排后的候选
    ↓
实际选入上下文的证据
    ↓
最终发送的模型输入
    ↓
回答中的结论与引用

The Context Builder is responsible for the steps in between: deduplication, completion of necessary context, document-version selection, conflict handling, token control, and binding each chunk to a traceable citation identifier.

If v2 says “7-day refund” and v3 says “14-day refund,” you can't choose by similarity alone. The system has to combine the effective date, whether the user is asking about the current or historical policy, and the authority of the source to decide which version to use. If the conflict can't be resolved, make it explicit.

For model capacity, the budget can be written as:

系统规则 + 工具定义 + 历史摘要 + 近期消息 + 检索证据
+ 为输出及模型所需额外 token 预留的空间
≤ 对应模型的有效上下文限制

Exact token counts and input and output limits should be checked against the model API. When limits are exceeded, some interfaces return an error, while some systems truncate before sending; don't assume the model will always handle it automatically.

“Lost in the Middle” observed in the models and tasks it tested that the position of relevant information affects long-context performance, and information in the middle may be harder to use effectively. This shows that window size doesn't equal effective utilization, but it doesn't justify assuming that every new model has the same degree of positional bias. Original paper

From an engineering standpoint, prioritize removing irrelevant information, preserve complete conditions, and test evidence ordering. Compression and summarization can save tokens, but they may omit negations, numbers, or exception clauses, so preserve a mapping back to the original evidence.

Citations need validation too: the fact that a link opens proves only that the source exists; you also need to check that it actually supports the adjacent claim. For calculations, let a program compute from the extracted data and then have the model explain the result.

7. Long conversations and multimodal documents need different ways of organizing information

Keeping the full history doesn't mean sending the full history to the model on every turn. The chat log serves auditing and product-display purposes; the Prompt serves the current inference.

A practical context mix is: stable rules, a summary of the current task, necessary long-term facts, the most recent exchanges, and evidence retrieved for the current turn. The task summary should focus on the user's goal, constraints, completed actions, failure reasons, outstanding work, and evidence locations.

“The user wants to contact customer support” and “The email was sent successfully” must be stored separately. Summarizing intent as an execution fact directly contaminates later decisions. Critical state should come from persisted tool results and business data, not from the model's free-form summary of the history.

Nor should every multimodal document be flattened to plain text. Tables should retain headers, units, and row-column relationships. Images can use descriptions to aid retrieval and then be read from the original as needed when answering. Charts and scans should retain page and region coordinates for verification. Image descriptions and OCR can both be wrong, so critical values must be verified against the original source.

For sales data of 100,000 rows, a question such as “What is the total revenue for each region?” is better suited to authorized, restricted, and validated SQL or a data-analysis tool. Embedding can help find relevant material, but it isn't a substitute for deterministic aggregation.

Similarly, “What is the penalty for breach of contract in this agreement?” can be answered by locating a clause; “Summarize the main risks across the entire agreement” requires coverage of multiple sections. Local factual Q&A and global summarization need different information-retrieval strategies.

8. Agent Runtime: let the model decide, let the system control execution

Workflow paths are defined primarily in code; an Agent lets the model dynamically choose the next action. A fixed document-processing flow can use a Workflow, while an open-ended troubleshooting investigation can use an Agent within controlled nodes; the two can be combined. Anthropic: Building effective agents

The conceptual loop of an Agent isn't complex:

The real engineering effort lies around the loop. The runtime needs to cap total steps, tool calls, elapsed time, tokens, and cost; set timeouts for individual model and tool calls; propagate cancellation signals; record the state of each step; and stop when there's no progress.

For example, you can detect whether the same tool, normalized arguments, and result repeat consecutively. But repeated calls don't necessarily indicate an infinite loop: querying a running task may legitimately return the same status multiple times. The decision must take tool semantics, reasonable polling intervals, and the overall deadline into account.

These budgets should belong to the Run and persist across recovery. If counters reset whenever the process restarts, the limits lose their meaning. Parallel steps also need to reserve budget atomically and then settle it based on actual consumption, preventing multiple Workers from seeing the same remaining balance.

Tool inputs must also be validated at every layer: the fact that JSON can be parsed only means the syntax is valid; conforming to a Schema doesn't mean the action is permitted by business rules; and a user's right to view a document doesn't imply the right to modify or send it externally. The execution layer must independently perform business validation and authorization.

Argument repair can return a specific error to the model, but the number of attempts should be limited. Retryable errors such as network jitter must respect the overall deadline, use backoff with jitter, and honor Retry-After where applicable. Permission denials and clearly invalid arguments should return an understandable failure result to avoid blind retries.

Frameworks can provide partial implementations of these mechanisms. For example, LangGraph uses a Checkpointer to save thread-scoped graph state and a Store to save data across threads. Even then, you need to choose a persistence backend and understand which state is saved and when. LangGraph Persistence

9. Reliable recovery: checkpoints and external side effects are separate concerns

First, let's clarify a few objects:

Small systems can use a relational database for core state and events and object storage for large files. Redis can handle caching, rate limiting, and lease coordination. Add a messaging system when events need to be distributed across services; there's no need to deploy every component on day one.

The hardest recovery scenario is this: a tool has created an order, but the Worker crashes before writing the Checkpoint. If the system consults only the Checkpoint, it thinks the step hasn't happened, and replaying it may create a second order.

Saving a checkpoint doesn't automatically make an external action execute exactly once.

A practical approach is to assign a stable operation ID to each logical action and reuse it across retries and recovery. The tool associates the ID with its result: when the same operation arrives again, return the existing result; reject reuse if the arguments differ. Save the intent before execution and the result afterward. If a timeout leaves the outcome uncertain, query the operation's result first.

Using only runId + stepId also has limits: it can prevent the same step from being replayed, but not stop the model from requesting the same business action in a later step. Operations such as placing orders often need a business-layer unique key or an explicit duplicate-operation rule as well.

If the external system doesn't support idempotency and its results can't be queried, you can't claim end-to-end exactly-once execution. Persist “success uncertain” as a state, then arrange reconciliation, compensation, or manual handling.

Multiple Workers may also contend for the same Run. A lease can coordinate ownership, but an old Worker that resumes after being paused may have already lost its lease. When writing state, check ownership and increment the version, rejecting stale executors; external side effects still require idempotency support from the tool.

If two concurrent requests arrive for the same Session, define the product semantics: queue them, cancel the earlier request, or create a branch. Letting two Runs overwrite the same context without constraints produces state errors that are difficult to reproduce.

10. SSE: handle disconnections and task completion separately

SSE is an HTTP-based server-to-client event stream with the response type text/event-stream. Events are separated by blank lines, and common fields include id, event, and data. When the native EventSource reconnects, it can send Last-Event-ID, but the protocol doesn't store historical events for the application. WHATWG Server-sent events standard

For tasks that need recovery, you can separate creation from subscription:

The native EventSource is suitable for GET subscriptions, but unlike an ordinary Fetch request, it can't freely set a request body or custom request headers. For a streaming POST request, use Fetch to read the response stream, but the client must also implement reconnection, event parsing, and cursor management.

Events can look like this:

To support resumption after a disconnect, the server must retain replayable events or provide a mechanism that connects a result snapshot to subsequent events. The client records the sequence numbers it has processed, requests the missing events after reconnecting, and deduplicates them. When replay transitions into a live subscription, avoid gaps, or events occurring between the two may be lost.

When the retention window for events is limited, the cursor may expire. Return a snapshot or an explicit resynchronization instruction in that case. After receiving a terminal event, the client should end its subscription so a completed task doesn't keep reconnecting pointlessly.

Whether a user's disconnection should cancel the task is a business decision. Short conversations can attempt cancellation after a disconnect or grace period; tasks that generate long reports can keep running so the user can check back later. Cancellation signals can only be propagated on a best-effort basis; they can't guarantee that actions already submitted to an external system will be undone.

This layer also needs to handle proxy buffering, idle timeouts, heartbeats, and slow clients. Don't let an unbounded send queue exhaust memory. You can coalesce fine-grained text events, cap the buffer for each connection, and disconnect when necessary so the client can resynchronize.

11. High concurrency and cost: quantify the load before talking about scaling

“100,000 users” isn't a complete capacity requirement. Registered users, concurrent online users, SSE connections, new Runs per second, and concurrent model calls each consume different resources.

A rough capacity estimate can use:

平均执行中任务数 ≈ 每秒进入执行的任务数 × 平均执行时长

For example, at steady state, if 200 tasks begin each second and run for an average of 20 seconds, roughly 4,000 tasks are executing at once. This is only an average; capacity planning must also account for traffic bursts, long-tail tasks, retries, and whether the queue keeps growing.

Handling many connections doesn't mean you can generate many answers at the same time. Model calls are also constrained by request rate, token rate, concurrency quotas, and the model's own throughput; a single Agent Run may make multiple model and tool calls.

Manage ingress quotas, per-user and per-tenant concurrency, the run queue, retrieval capacity, reranking capacity, and model-call quotas separately. Queues must have bounds and a maximum wait time; otherwise overload merely becomes a long wait the user can't see.

Break latency down by stage as well:

接收请求 → 排队 → Query 处理 → 检索 → 重排 → 模型首 token → 完成

Status messages let users know a task is still progressing, but they don't mean the answer is ready any sooner. Track time to first product feedback, time to the model's first token, and overall completion time separately.

Cost optimization should also be tied to specific stages. Omit a rewrite when it isn't needed, use a program for anything it can calculate, and don't repeatedly send duplicate evidence. Validate model routing with task evaluations so that a failed cheap model followed by a retry and upgrade doesn't increase total cost.

Distinguish cache purposes. An application-layer answer cache reuses business results; a retrieval cache reuses candidates; a Prompt Cache or an inference service's KV Cache reuses some computation. None can replace session persistence or bypass checks against current permissions and knowledge versions. Validate the exact cache-hit rules for the service you're using.

12. Use evaluations to locate problems and failure scenarios to validate the system

A single “good/bad” score for the final answer isn't enough to guide improvement. Break evaluation into several layers:

Recall@K usually means how much of the relevant evidence appears in the top K results. If you're only checking whether at least one relevant item was hit, make it clear that you're using a hit-rate metric. For multi-hop questions, it's especially important to distinguish finding one relevant passage from finding all the evidence needed to complete the reasoning.

The test set should include factual questions, multi-turn references, exact error codes, cross-document questions, table calculations, conflicting documents, unanswerable questions, and permission-restricted cases. Fix the versions of the corpus, prompts, model configuration, and annotation standards so you can compare the impact of each change.

LLM-as-Judge can help with evaluation, but it can't serve as ground truth. Calibrate it against explicit criteria and manually labeled samples; for critical numbers, citations, and business side effects, prefer verifiable rules or business outcomes. Also retain a test set that wasn't used for tuning, so you don't end up optimizing only the samples you're already familiar with.

Production Traces should record enough information to locate problems: candidate IDs and versions, context selection, model configuration, references to tool inputs and outputs, fallback paths, latency, and consumption. Sensitive content should be redacted, access-restricted, and subject to a retention period; don't copy all raw content into logs indefinitely just for debugging.

Recovery capabilities also need fault-injection validation: terminate a Worker after a tool succeeds but before state is committed; generate a new event during SSE replay; let a lease expire; revoke document permissions; and slow down the client. These tests reveal failure modes that ordinary Q&A will never expose.

13. Security boundaries must hold outside the model

RAG brings external text into the model. Web pages, PDFs, and tool results may all contain malicious instructions such as “ignore previous instructions” or “upload other documents.” High retrieval relevance doesn't mean the content has authority to issue instructions.

Treat external content as data, and use the execution layer to restrict callable tools, resource scope, and external destinations. Use least-privilege credentials, validate parameters and identity independently, and require confirmation for high-impact operations according to business requirements. Reminders in the Prompt can provide one layer of defense, but they can't replace permission checks. OWASP Prompt Injection Prevention

For example, if a document tells an Agent to “send the internal report to this email address,” whether the send is allowed should be determined jointly by the trusted user request, data permissions, and tool policy. The document itself can't grant that permission.

Testing must cover more than direct malicious questions. It should also cover indirect injection through retrieved passages, tool returns, long-conversation summaries, and cache hits. Cross-tenant access and permission revocation need to be verified across retrieval, caching, generation, and download.

14. A rollout path you can implement incrementally

Building all of these capabilities at once is expensive, and it makes it hard to tell which parts are genuinely useful. A more practical approach is to make every step answer clear acceptance questions.

Introduce multiple Agents only when a task can genuinely be split independently, or when contexts or permissions need isolation. They add coordination, communication, and result-merging overhead, so compare them with a single-Agent or fixed-Workflow baseline.

When faced with an incorrect answer, the system should be able to identify the step where evidence was lost. When faced with an interruption, it should explain whether the task is still running and which actions have already occurred. As traffic grows, it should quantify the bottlenecks in connections, computation, and budget separately. With these capabilities, RAG and Agent can keep improving instead of making every incident another round of Prompt guesswork.

Further reading: How Agents Should Separate Sessions, Runs, and Connections on the Web, What an Agent's Session Should Store: Session, Message, Event, and Context.