From RAG to Agent: Engineering Design for Production-Grade AI Systems

A knowledge base Q&A demo typically only requires three steps: chunk documents, retrieve fragments, and call the model. Once you go live, the problems become much more specific: why does the model get the answer wrong when it's clearly in the document? Why does the task disappear when the user refreshes the page? Can we retry directly after a tool request times out? When concurrent users increase, what gives out first—the connections, the database, or the model quota?

These issues span information retrieval, distributed systems, and model application. Memorizing terms like RAG, Rerank, and Checkpoint individually isn't enough to explain how they relate to each other.

This article uses an enterprise knowledge base assistant as an example: users upload documents, engage in multi-turn questioning about those documents, and when necessary, have the system query data, call tools, and generate reports. Following this task, we explain how retrieval provides evidence, how context organizes evidence, how Agents推进 execution, and how the system continues working after failures.

The candidate counts, budgets, and capacity numbers in this article are examples used to explain the design—they need to be evaluated and load-tested with real data and shouldn't be used directly as production configurations.

1. First, Distinguish Knowledge Flow, Execution Flow, and State

The system can be divided by responsibilities as shown below. The modules in the diagram represent logical boundaries; small projects can easily deploy everything in a single service.

Document Upload → Parsing & Cleaning → Structured Chunking → Full-Text Index / Vector Index
                                        ↑
User → API / Auth → Create Run → Retrieval & Context Construction → Model
                         │                       │
                         │             Tool Call ←┘
                         │                │
                         │         Save Result, Continue Execution
                         ↓
                 Run State / Checkpoint / Event
                         ↓
                    SSE → Client

Here are two easily confused facts.

First, RAG and Agent can be combined but don't require each other. A fixed "retrieve then answer" can be one Workflow; an open-ended investigation task can have the Agent decide what to query next, with retrieval being just one of its tools.

Second, even if a single model call only depends on the messages passed in, the entire application still has state. Who the user is, where the current task execution stands, which tools have succeeded, and how much budget remains must all be managed by the system. The model context is only one part of this.

Therefore, when designing, first answer three questions: where does knowledge come from, how does the task progress, and where are execution facts stored. Component selection comes after answering these questions.

2. Knowledge Ingestion: Parsing Quality Determines the Ceiling of Retrieval

The first step in document ingestion is parsing. Converting a PDF directly into one long string may lose heading hierarchies, table relationships, image captions, and page numbers. Even with powerful Embedding, you can't recover facts that have already been lost.

A more appropriate intermediate representation is "document—section—content block." Content blocks can be paragraphs, tables, images, or code, while preserving source location. A chunked fragment can include:

{
  "chunkId": "refund-v3-section2-01",
  "documentId": "refund-policy",
  "documentVersion": 3,
  "sectionPath": ["After-Sales Policy", "Refund Conditions"],
  "page": 12,
  "content": "Refunds can be applied for within 14 days of purchase if the product has not been activated.",
  "effectiveFrom": "2026-01-01",
  "tenantId": "tenant-a",
  "aclRef": "policy-sales"
}

The body text is used for retrieval and answering; document versions handle新旧 conflicts; page numbers and positions are used for citations; tenant and permission information is used for access control. These should be established during ingestion.

Cleaning should also be restrained. Repetitive headers and navigation bars are usually noise, but units in tables, negations in contracts, and exception conditions in footnotes may directly affect the answer. The goal of cleaning is to remove interference while preserving semantics.

The same knowledge can establish both full-text and vector indexes. When updating documents, you also need to consider when both indexes become visible: if the full-text index is already v3 while the vector index is still v2, a single query might mix contradictory versions.

One viable design is to assign versions to index builds and only switch to queryable versions after completion, keeping each candidate with its document version. Deletions and permission revocations must also synchronously affect indexes and caches, not just modify the original files.

3. The Key to Chunks: Retrieval Granularity and Semantic Completeness

Fixed-length chunking is easy to implement, but it doesn't understand whether a sentence is complete or the relationship between rules and exceptions.

For example, refund rules contain two conditions: the purchase must be within the time limit and the product must not be activated. If the chunk boundary falls right between them, retrieving only one fragment could lead to incorrect conclusions.

Different chunking methods serve different purposes:

Overlap can mitigate boundary issues but increases storage, recall duplication, and context overhead. It also can't guarantee that distantly separated clauses are retrieved together.

For clearly structured long documents, you can use Parent–Child Retrieval: use smaller fragments for retrieval, then supplement the containing section or necessary adjacent blocks upon hits. This maintains positioning precision while preserving the context needed for answering. The supplement scope is still constrained by token budget—you can't stuff an entire chapter to the model just because one hit occurred.

When documents reach hundreds of thousands of characters, you can also build hierarchical indexes of document summaries, chapter summaries, and original fragments. Hierarchical routing can narrow the search scope, but if the first step chooses the wrong chapter, there's no chance to find the answer later. Therefore, it's best to retain fallback paths for cross-chapter or global search.

The granularity used for retrieval can be smaller than the granularity provided when generating answers. This distinction is more useful than fixing a single chunk size.

4. Hybrid Retrieval: Letting Exact Matching and Semantic Matching Complement Each Other

When a user asks "what to do when the computer freezes," but the document says "how to handle unresponsive computers," vector retrieval may recognize the semantic connection. When users input ORA-00942, order numbers, or API names, term matching is often more direct.

The main factors in BM25 include term frequency, inverse document frequency, and document length normalization. Term frequency contribution gradually saturates; rarer words typically have higher discriminative power; length normalization mitigates the problem where long documents dominate simply through term frequency. The actual effect of Lucene BM25Similarity documentation also depends on the tokenizer, field settings, and index granularity—you can't assume "using BM25" guarantees exact matching success.

Vector retrieval maps queries and content into vectors, finding candidates by distance or similarity. Large-scale retrieval typically uses approximate nearest neighbor algorithms, requiring trade-offs among recall rate, latency, and index cost. Query vectors and document vectors must follow the same encoding convention of the retrieval model; when the model upgrades, you also need to plan for re-embedding and index switching.

When merging two result sets, you shouldn't directly add raw scores together. BM25's 12.8 and vector similarity's 0.86 don't have inherently consistent scales. You can calibrate scores on evaluation data, or first use rank-based Reciprocal Rank Fusion:

RRF(d) = Σ 1 / (k + rank_i(d))

Here the sum only covers result lists where document d appears, ranks start from 1, and k is a smoothing constant. For example, with k = 60, a result ranked 1st and 2nd in two paths gets a score of 1/61 + 1/62. RRF avoids direct alignment of different raw score scales but loses rank difference information and isn't guaranteed optimal on all datasets. Elasticsearch RRF documentation

A complete flow can be:

Current Question + Necessary History
        ↓
Query Rewrite as Needed (De-reference, Reformulate)
        ↓
Retrieval with Tenant and Permission Constraints
        ├── BM25 Top 50
        └── Vector Top 50
                 ↓
           Merge, Deduplicate, RRF
                 ↓
             Candidates Top 30
                 ↓
               Rerank
                 ↓
         Context Selection & Citation Binding

Query rewriting isn't required every time. When a user asks "why did it get slower," the object needs to be filled in from history; when the user already provides a complete error code, additional rewriting may increase latency or even alter key terms. The original question should be preserved, along with what was changed during rewriting.

Permission filtering must be generated server-side based on trusted identity and pushed down to both retrieval paths as much as possible. Recheck access authorization before passing content to external reranking services or models. Passing unauthorized content to the model first and only hiding citations at the end is already too late.

5. Rerank Improves Ranking Precision but Also Becomes a Latency Source

Common Bi-Encoders encode Query and document separately, so document vectors can be pre-computed. Cross-Encoder reads Query and candidate content together, scoring relevance through interaction between the two—suitable for reranking a small number of candidates. It needs to process query-candidate pairs one by one, making it unsuitable for direct exhaustive search on large-scale corpora. Sentence Transformers: Retrieve & Re-Rank

This is also why we "retrieve first, then rerank." Rerankers can only improve candidates that have already been retrieved; they can't find evidence missed by the earlier stage.

If Rerank is too slow, first look at candidate count, candidate length, and queue wait times. You can reduce duplicate candidates, adjust quantity based on question difficulty, limit obviously irrelevant long texts, then evaluate batch processing, smaller models, quantization, or inference backend optimizations.

Fewer candidates isn't always better. Premature pruning loses evidence; overly large batch processing may improve throughput but make low-traffic requests wait longer. Optimization should observe both retrieval quality and tail latency.

Cache keys should at least distinguish Query, document content version, and reranker model version. When a request gets a cached result, you still need to perform current permission checks.

Rerank timeout can fall back to hybrid retrieval ranking, but this is a quality degradation. Regular information queries may accept this; for workflows with strict evidence requirements, it may be better to fail explicitly or request additional information. Degradation strategies should be recorded in traces to later explain quality changes.

6. After Retrieval Hits, Evidence Must Actually Make It into the Answer

"Correct recall, wrong answer" isn't contradictory. Correct fragments may drop out of candidates after reranking, get truncated during prompt assembly, enter context along with outdated documents, or be misinterpreted by the model.

When debugging, you need to see these sets:

Candidates Retrieved
    ↓
Candidates After Reranking
    ↓
Evidence Actually Included in Context
    ↓
Final Model Input Sent
    ↓
Conclusions and Citations in the Answer

The Context Builder's responsibility is to complete the intermediate steps: deduplication, filling necessary context, selecting document version, handling conflicts, controlling tokens, and binding traceable citation identifiers to fragments.

If v2 says "7-day refund" and v3 says "14-day refund," you can't just choose by similarity. The system needs to decide which version to use based on effective time, whether the user is asking about current rules or historical rules, and source authority. When conflicts can't be resolved, they should be explicitly presented.

For model capacity, the budget can be written as:

System Rules + Tool Definitions + History Summary + Recent Messages + Retrieved Evidence
+ Space Reserved for Output and Model-Required Additional Tokens
≤ Effective Context Limit for Corresponding Model

Specific token counts and input/output limits should be checked against the model interface. When limits are exceeded, some interfaces throw errors while some systems truncate before sending—you can't assume the model will always handle it automatically.

"Lost in the Middle" observed in its tested models and tasks: the position of relevant information affects long-context performance, with middle positions potentially harder to utilize effectively. This shows window length doesn't equal effective utilization capacity, but you can't conclude from this that all new models have the same degree of positional bias. Paper

In engineering, prioritize reducing irrelevant information, preserve complete conditions, and test evidence ordering. Compression and summarization save tokens but may also miss negations, numbers, or exception clauses, so maintain mappings to original evidence.

Citations also need verification: a link opening only proves the source exists; you also need to check if it actually supports adjacent conclusions. When calculations are involved, you can have programs operate on extracted data first, then have the model explain the results.

7. Long Conversations and Multimodal Documents Require Different Organization Methods

Saving complete history doesn't mean sending complete history to the model every turn. Chat history serves audit and product display purposes, while prompts serve current reasoning.

A usable context combination is: stable rules, current task summary, necessary long-term facts, recent conversation turns, and current retrieval evidence. The task summary should focus on user goals, constraints, completed actions, failure reasons, pending items, and evidence locations.

"User wants to contact the customer" and "email sent successfully" must be saved separately. Summarizing intent as execution facts will directly pollute subsequent decisions. Key state should come from persisted tool results and business data, not the model's free summarization of history.

Multimodal documents also shouldn't all be flattened to plain text. Tables should preserve headers, units, and row/column relationships; images can use descriptions to aid retrieval, with original images read on-demand during answering; charts and scanned documents should preserve page numbers and region locations for verification. Image descriptions and OCR can both be wrong—key numbers need verification against original sources.

For 100,000 rows of sales data, if the question is "what's the total revenue by region," SQL or data analysis tools with authorization, limits, and validation are more suitable. Embedding helps find relevant materials but isn't suitable for replacing deterministic aggregation calculations.

Similarly, "what's the penalty in this contract" can locate the clause; "summarize the main risks of the entire contract" requires covering multiple chapters. Local factual Q&A and global summarization should use different information retrieval strategies.

8. Agent Runtime: Let the Model Decide, Let the System Control Execution

Workflow paths are mostly pre-defined in code; Agents let the model dynamically decide the next action. Fixed document processing flows can use Workflows, while open-ended fault investigation can use Agents within controlled nodes—the two can be combined. Anthropic: Building effective agents

The conceptual cycle of an Agent isn't complicated:

Load State → Construct Context → Call Model
                         ├── Final Answer → End
                         └── Tool Request
                                ↓
                     Parameter Validation, Permission Check, Budget Check
                                ↓
                           Execute and Save Result
                                ↓
                            Enter Next Round

The real engineering work is around the cycle. Runtime needs to limit total steps, tool call counts, time, tokens, and cost; set timeouts for individual model calls and tool calls; propagate cancellation signals; record each step's state; and stop when no progress is made.

For example, you can detect whether the same tool with normalized parameters and results repeats consecutively. But repeated calls aren't necessarily infinite loops: querying a running task may return the same status multiple times by design. Judgment must combine tool semantics, reasonable polling intervals, and overall deadlines.

These budgets should belong to the Run and persist through recovery. If process restart resets the count, the limits become meaningless. Parallel steps also need atomic budget reservation, then settlement based on actual consumption, to prevent multiple workers from seeing the same balance simultaneously.

Tool input must also be checked layer by layer: JSON parsing only means syntax is correct; conforming to Schema doesn't mean business allows it; a user有权 viewing a document doesn't mean they can modify or export it. The execution layer needs to independently perform business validation and authorization.

Parameter fixing can return specific errors to the model but should limit the number of attempts. Retriable errors like network jitter should respect total deadlines, use randomized backoff, and respect Retry-After when applicable. Permission denials and clearly illegal parameters should return understandable failure results to avoid blind retrying.

Frameworks can provide partial implementations of these mechanisms. For example, LangGraph uses Checkpointer to save graph state within a thread and Store to save data across threads; when using them, you still need to choose a persistence backend and understand what state is saved and when. LangGraph Persistence

9. Reliable Recovery: Checkpoints and External Side Effects Are Two Different Things

Let's first clarify several objects:

Small systems can use relational databases for core state and events, and object storage for large files. Redis can be used for caching, rate limiting, and lease coordination. Only add message systems when cross-service distribution is needed—no need to deploy all components from day one.

The hardest recovery scenario is: a tool has created an order, but the Worker crashes before writing the Checkpoint. The system, looking only at Checkpoint, would think this step hasn't been done; re-executing might create a second order.

Saving checkpoints doesn't automatically make external actions execute only once.

Viable approaches include assigning stable operation IDs to logical actions and reusing them during retry and recovery. The tool side associates this ID with the result: when the same operation arrives again, return the existing result; reject reuse when parameters don't match. Save intent before execution, save result after execution; when encountering timeout uncertain states, prioritize querying the operation result.

Using only runId + stepId also has boundaries: it can prevent the same step from being replayed, but it can't prevent the model from proposing the same business action in the next step. Orders and similar operations often need business-layer unique keys or explicit duplicate operation rules.

If the external system doesn't support idempotence and can't query results, you can't claim to have achieved end-to-end exactly-once. You need to save "uncertain whether successful" as a state and arrange reconciliation, compensation, or manual processing.

Multiple Workers may also compete for the same Run. Leases can coordinate ownership, but when an old Worker resumes after pausing, it may have already lost the lease. When writing state, check ownership and incremental version, reject expired executors; external side effects still need tool-side idempotence cooperation.

When the same Session receives two concurrent requests, product semantics should also be defined: queue them, cancel the previous one, or create branches. Letting two Runs unconstrained overwrite the same context produces hard-to-reproduce state errors.

10. SSE: Handle Connection Disconnect and Task End Separately

SSE is an HTTP-based server-to-client event stream with response type text/event-stream. Events are separated by blank lines, with common fields id, event, and data. Native EventSource can send Last-Event-ID on reconnect, but the protocol doesn't save historical events for the application. WHATWG Server-sent events standard

For tasks that need recovery, you can separate creation and subscription:

POST /runs
    → Create task, return runId

GET /runs/{runId}/events
    → Subscribe to execution events

GET /runs/{runId}
    → Query current status and results

POST /runs/{runId}/cancel
    → Explicit cancellation request

Native EventSource is suitable for GET subscriptions and can't arbitrarily set request body or custom headers like regular Fetch. When you need POST streaming requests, you can use Fetch to read the response stream, but reconnection, event parsing, and cursor management also need client-side implementation.

Events can be expressed as:

id: 101
event: answer.delta
data: {"runId":"run-42","text":"Refund conditions include"}

id: 102
event: run.completed
data: {"runId":"run-42","resultId":"result-42"}

To implement resumable connections, the server must save replayable events, or provide result snapshots with mechanisms to bridge to subsequent events. The client records processed sequence numbers, requests deduplication and replenishment after reconnecting. When replay transitions to live subscription, gaps must be avoided—otherwise events happening exactly between the two will be lost.

When event retention is limited, cursors may expire. In this case, return a snapshot or explicit resync instruction. After receiving a termination event, the client should end the subscription to avoid pointless reconnections for completed tasks.

Whether user disconnect cancels the task depends on business decision. Short conversations can try to cancel after disconnect or a grace period; tasks generating long reports can continue running, letting the user query later. Cancellation signals can only be best-effort—they can't guarantee actions already submitted to external systems are rolled back.

This layer also needs to handle proxy buffering, idle timeouts, heartbeats, and slow clients. Don't let infinitely growing send queues exhaust memory. You can merge fine-grained text events, set buffer limits on connections, and disconnect when necessary, letting the client resync.

11. High Concurrency and Cost: Quantify Load First, Then Talk About Scaling

"Hundred thousand users" isn't a complete enough capacity requirement. Registered users, concurrent online users, SSE connections, new Runs per second, and concurrent model calls each consume different resources.

A rough capacity estimate can use:

Average In-Flight Tasks ≈ Tasks Entering Execution per Second × Average Execution Duration

For example, in a steady state, 200 tasks start per second with average execution duration of 20 seconds, giving approximately 4,000 in-flight tasks. This is only the average—capacity planning also needs to consider burst traffic, long-tail tasks, retries, and whether queues are continuously growing.

Holding many connections doesn't mean having the ability to generate many answers simultaneously. Model calls are also constrained by request rate, token rate, concurrent quota, and their own throughput; an Agent Run may call the model and tools multiple times.

You should separately manage entry quotas, user and tenant concurrency, run queues, retrieval capacity, reranking capacity, and model call quotas. Queues must have boundaries and wait limits, otherwise overload just becomes long waits that users can't see.

Latency also needs to be observed in segments:

Receive Request → Queue → Query Processing → Retrieval → Rerank → Model First Token → Complete

Status updates let users know the task is still progressing, but that doesn't mean the answer is coming faster. Product-level time to first feedback, model time to first token, and overall completion time should be recorded separately.

Cost optimization should also fall to specific stages. Skip rewrite if it can be omitted; delegate what can be calculated to programs; don't repeatedly send duplicate evidence; model routing must be validated with task evaluation to avoid cheap models failing and retrying, which increases total cost.

Caching needs to distinguish usage. Application-layer answer caching reuses business results; retrieval caching reuses candidates; Prompt Cache or inference service KV Cache reuses partial computation. None of these can replace session persistence or bypass current permission and knowledge version checks. Specific cache hit rules should be verified against the services used.

12. Use Evaluation to Locate Problems, Also Use Failure Scenarios to Verify the System

A single "good/bad" score for the final answer isn't enough to guide improvement. Evaluation can be broken into layers:

Recall@K typically means how much of relevant evidence enters the top K results. If you're just checking "whether at least one hit," that should be explicitly stated as a hit rate metric. Multi-hop questions especially need to distinguish finding one relevant fragment from finding all evidence needed to complete reasoning.

Test sets should include factual questions, multi-turn references, exact error codes, cross-document questions, table calculations, document conflicts, unanswerable cases, and permission-restricted cases. Using fixed versions of corpus, prompts, model configurations, and labeling standards is necessary to compare changes from a single adjustment.

LLM-as-Judge can assist evaluation but shouldn't be used directly as ground truth. Use explicit standards and human-annotated samples for calibration; for key numbers, citations, and business side effects, prioritize verifiable rules or business results. Also keep test sets that didn't participate in tuning to avoid only optimizing familiar samples.

Online traces should record enough information to locate problems: retrieval candidate IDs and versions, context selection, model configuration, tool input/output references, degradation paths, latency, and consumption. Sensitive content should be de-identified, access-limited, and have retention limits—don't copy all raw text to logs for debugging indefinitely.

Recovery capability also needs fault injection verification: terminate Worker after tool success but before state submission; inject new events during SSE replay; expire leases; revoke document permissions; slow down clients. These tests can discover vulnerabilities that normal Q&A would never expose.

13. Security Boundaries Must Be Established Outside the Model

RAG brings external text into the model. Web pages, PDFs, or tool results may all contain malicious content like "ignore previous instructions" or "upload other documents." High retrieval relevance doesn't mean the content has command authority.

External content should be treated as data, and the execution layer should limit callable tools, resource scope, and export destinations. Use least-privilege credentials, independently verify parameters and identity, and set confirmation mechanisms for high-impact operations that meet business requirements. Reminders in prompts can serve as one layer of protection but can't replace permission checks. OWASP Prompt Injection Prevention

For example, if a document tells the Agent "send the internal report to this email," whether sending is allowed should be determined by trusted user requests, data permissions, and tool policies combined. The document itself can't grant this permission.

Testing also shouldn't only cover directly malicious questions—also cover indirect injection through retrieved fragments, tool returns, long conversation summaries, and post-cache-hit scenarios. Cross-tenant access and permission revocation need verification across retrieval, caching, generation, and download stages.

14. A Phased Implementation Path That Can Be Progressively Adopted

Building all the above capabilities at once is very costly and makes it difficult to judge which parts are actually useful. A more practical advancement approach is to have clear acceptance questions for each step.

Multiple Agents can be introduced only when tasks can truly be independently split, context needs isolation, or permissions need separation. It adds state coordination, communication, and result merging costs—also compare with single Agent or fixed Workflow baselines.

Facing a wrong answer, the system should be able to point out where evidence was lost in which step; facing an interruption, it should be able to explain whether the task is still running and what actions have occurred; facing growing traffic, it should be able to quantify bottlenecks in connections, computation, and budget respectively. Only after having these capabilities can RAG and Agent iterate continuously instead of starting to guess at prompts with every failure.

Further Reading: After Agent Enters Web, How Should Session, Run and Connection Be Separated, What Should Agent Conversation Save: Session, Message, Event and Context.