RAG System Architecture: Preprocessing, Retrieval & Agentic Design

Parsing Raw Documents

Real document corpora rarely arrive as clean text. Scanned paperwork comes back as OCR JSON (Google Vision, AWS Textract, Azure Document Intelligence) with text reconstructed from pages → blocks → paragraphs → words → symbols, each carrying its own confidence score. The parser's job is to walk that structure, reconstruct readable text per block, and compute a mean confidence per block. That confidence score becomes a first-class field that follows the chunk all the way to the final answer, not a debug log line nobody reads.

Document type classification runs before chunking, since chunking strategy depends on it. A two-pass heuristic works well in practice: check filename keywords first, then fall back to content keywords in the first few hundred characters. It has to tolerate OCR corruption too. A national ID scanned badly enough to read "CARTE MATIONALE" instead of "CARTE NATIONALE" should still classify correctly through fuzzy keyword matching, not an exact string match that quietly fails on every third scan.

Chunking Strategy by Document Class

The biggest mistake in RAG preprocessing is applying one uniform chunk size to every document. A fixed-size splitter (e.g, 500 tokens with overlap) is a reasonable default for unstructured prose. But it cuts through the middle of clauses, line items, and structured fields, which is exactly the content people ask precise questions about. Chunk by document structure instead of character count, and the citation unit stays intact:

Fallback cascade: header-based regex chunking fails whenever a document doesn't match the expected template — a different contract format, a report missing its usual headers, that sort of thing. The pipeline needs a graceful degradation path. If header regex matches nothing, fall back to OCR block boundaries from the source layout. If that also comes up empty, fall back to treating the whole document as one chunk. A chunking failure should degrade retrieval quality. It should never crash ingestion.

Citation-stable IDs: every chunk carries a stable, human-readable ID encoding its exact location, like invoice_2024_0143#line_items or contract_A118#section_12. This ID is the unit the retrieval layer returns and the unit the solver is instructed to cite. Without it, "the answer came from somewhere in this 40-page document" is the best a system can do; with it, the answer points to an exact, verifiable section.

Structured Extraction

Chunking solves retrieval; it doesn't solve structured questions like "which documents are missing" or "is this certificate expired." Those need a second pass: one LLM call per document that reconstructs the full document text from its chunks and extracts a structured JSON profile, using a prompt tailored to that document type's expected fields (parties, amounts, dates, expiry flags, status).

Cache aggressively. Profiles are stable between corpus updates. Re-extracting on every request costs one LLM call per document, per query, which doesn't scale. Extract once at ingestion, write the result to disk keyed by document ID, and skip any document that already has a cached profile on subsequent runs. The upgrade path, once the corpus becomes dynamic (new documents arriving continuously), is incremental extraction triggered on document arrival rather than a full batch re-run.

Handling Sensitive Data

Most write-ups on RAG skip this step entirely, which is odd given how much of a real corpus is exactly the kind of data you don't want sitting in a vector index unprotected. Full account numbers, payment card numbers, national ID numbers, medical record numbers. None of that belongs anywhere near a retrieval layer that a poorly-scoped prompt, an overly broad tool call, or a compromised vector store could expose.

The fix is a redaction step between extraction and everything downstream of it:

1. Run a PII detector over every document before chunking, not after. Sensitive spans that get split across chunk boundaries are much harder to catch reliably.
2. Redact or tokenize sensitive fields (full PANs, account numbers, SSNs), replacing each with a stable placeholder token rather than a blackout mark, so downstream extraction can still reason about "an SSN was present here" without ever seeing the value.
3. Store the token → real value mapping in an encrypted vault, kept entirely separate from the retrieval layer. If the vector store or an LLM's context window is ever exposed, the mapping isn't sitting next to it.
4. Only clean, tokenized text should ever reach the vector DB or an LLM prompt. A chunk that fails tokenization for any reason should fail closed: held for manual review, never indexed as-is.

This is the one part of the architecture I'd fight hardest to keep a hard requirement instead of a config flag someone quietly leaves off in staging. A mediocre retrieval score is a Tuesday-afternoon problem. A real SSN sitting in a vector index because someone skipped the redaction step is the kind of mistake that ends the project. Or worse.

Why Embeddings Alone Aren't Enough

Embeddings give the system semantic search: matching a query to a chunk by meaning rather than exact words. That matters for three concrete reasons that show up in every document-heavy vertical:

• Users don't phrase questions in the document's words. A user asks "what's the sale price?" but the contract says "for the principal sum of $285,000." There's no shared keyword — but in embedding space, both land in the same semantic neighborhood.
• Source text is unreliable. Scanned documents contain OCR corruption. Keyword matching breaks on garbled tokens; embeddings degrade more gracefully because surrounding context still places the chunk roughly correctly.
• Domain synonyms and paraphrase. "Buyer", "purchaser", "the acquiring party" — a good embedding model maps these near each other without anyone maintaining a synonym list by hand.

But embeddings are weak at exactly the things lexical search is strong at: account numbers, article references, product SKUs, proper names, legal codes. A query for "clause 14.3" should retrieve the chunk containing that literal string with certainty. An embedding model has no obligation to rank it first. This is the core justification for hybrid retrieval, covered next.

Embedding Model & Vector Storage

For a self-hosted stack, a local sentence-transformers model (e.g. a multilingual MiniLM variant) producing normalized vectors is a solid default: it's free per-call, runs on CPU, and caches its weights locally after first load. A hosted embedding API (OpenAI, Cohere, Voyage) trades that infrastructure cost for simpler ops and, in some cases, stronger retrieval quality. Worth revisiting if precision becomes the bottleneck at scale.

Vector storage follows the same right-sizing logic. A numpy array with a brute-force dot product (cosine similarity, since vectors are normalized) comfortably serves corpora up to the low tens of thousands of chunks, in-memory, with zero service dependency. A dedicated vector database (Pinecone, Qdrant, pgvector) earns its operational cost once the corpus grows past that point, or once filtered approximate nearest-neighbor search at consistent low latency becomes a hard requirement.

Three-Stage Hybrid Search

Cosine-only retrieval is the most common production shortcoming in RAG systems. It misses exact lexical matches. The fix is a three-stage pipeline that fuses semantic and lexical retrieval, then reranks the result with a model that actually looks at query and document together:

The cross-encoder's output logit, passed through a sigmoid, is a better-calibrated relevance score (0-1) than raw cosine similarity, and doubles as a useful per-request metric. The highest score across a request's retrieval calls is a solid proxy for retrieval_relevance. Track it over time; a drop signals a corpus or query-drift problem before users start complaining.

Metadata filters (document type, collection, date range) should apply before ranking, not after. Filtering a top-k result set after the fact just means some queries silently return fewer than k results. Both the cosine and BM25 stages should accept the same filter and apply it to the candidate set up front.

The Loop-Depth Problem

The default agent pattern most people reach for is ReAct: interleaved reasoning and tool calls, where each step can adapt based on what the previous step found:

ReAct is genuinely adaptive, but it's a true loop. Cost and latency scale with however many steps the model decides it needs, and it requires a hard max_iterations guard to prevent runaway depth on an ambiguous query.

ReWoo: Plan Once, Execute, Solve

ReWoo (Reasoning Without Observation) produces the full tool plan upfront in a single pass, executes every tool (in parallel where possible), then synthesizes:

This is always exactly two LLM calls, one to plan and one to solve, regardless of query complexity. Tool execution happens in between, and it's plain deterministic code, not a model call. The loop-depth problem is solved architecturally: there is no loop. The only guard needed is a cap on how many tool calls a single plan may contain (e.g. max_tools_per_plan = 5), which prevents an overly ambitious plan from burning excess tokens. That's a simpler and more principled constraint than a runtime iteration counter.

Why this fits most document-QA workloads: query types in a bounded document corpus are more predictable than they first appear. "Are there inconsistencies in this collection?" almost always maps to fetching every document in that collection; "what's missing?" almost always maps to the inventory tool. The planner can determine the right tools upfront without needing to see intermediate results first.

ReWoo vs. ReAct: Trade-offs

ReAct remains the right upgrade path once query patterns become genuinely open-ended, where an intermediate tool result needs to change the retrieval strategy mid-query rather than just supply data to a fixed plan.

Tools: Keep Them Deterministic

Every tool the planner can call should be pure, deterministic Python with no LLM calls inside it. The planner decides what to run, the solver decides what it means, and the tool layer just executes, in-memory, sub-millisecond per call:

The get_document_inventory tool is worth calling out: it returns a completeness checklist (which document types are required per collection, and which are absent) computed directly from the extracted profiles, so the solver can answer "what's missing" without inferring absence from a list. That's a task LLMs are surprisingly unreliable at.

Source Citation & Confidence Flagging

Every chunk's stable ID travels with it from retrieval through to the final answer. The solver's system prompt should require citing exactly which chunk IDs it used. That turns every answer into something a human can verify against the source in seconds:

For chunks below an extraction-confidence threshold, the agent should flag this explicitly rather than presenting uncertain data as fact. That's non-negotiable in regulated verticals:

What Good Observability Actually Looks Like

In practice this means every response carries its own receipt. A response payload from a system I've run in production looks roughly like this:

That single object answers "is this response trustworthy" and "what did it cost" without anyone having to go dig through logs. Quality itself splits into three separate things worth tracking separately, because they fail for different reasons and get fixed in different layers:

• Retrieval relevance. Always on, free to compute, comes straight out of the reranker. Are the retrieved chunks actually close to the query?
• Faithfulness. Is the answer grounded in what was retrieved, or is the model filling gaps on its own? This is the one that matters most anywhere a wrong answer has real consequences, which is most document-heavy verticals.
• Answer relevance. Usually scored offline against a held-out set of queries. Does the answer actually address what was asked, independent of whether it's grounded?

Keep evaluation logic decoupled from the live request path. Scoring approaches change as you learn more about where the system fails; the pipeline serving user traffic shouldn't have to change every time the eval rubric does.

Testing What You Can't Eyeball

Retrieval bugs don't throw exceptions. A hybrid search that quietly starts returning the wrong chunks still returns a response, and it can look fine right up until someone notices the answers have gotten worse. That makes automated testing of the retrieval layer non-negotiable, not a nice-to-have.

Mock the LLM calls in tests. Run the deterministic pieces (parsing, chunking, hybrid search, tool execution) against real fixture data instead. A parser change or a ranking tweak should never need a live model call to verify. On top of that, keep a small set of representative test queries per corpus, with the chunk IDs each one is expected to retrieve, and run that set before every deploy. It's a handful of test cases, not a research benchmark, but it catches the regression that code review alone won't.

One failure mode worth testing for specifically: a fact that spans two adjacent chunks and gets missed because retrieval only surfaces one of them. Section-level chunking mostly avoids this, but not always. The workaround at query time is prompting the solver to consider adjacent sections when a claim looks incomplete. The more durable fix is a small sliding-window overlap between adjacent chunks, so the boundary itself stops being a place where information gets lost.

Manual scoring against a handful of test questions is a fine starting point and works for a while. Once query volume and corpus size outgrow what a person can spot-check, a framework like RAGAS for automated faithfulness and relevance scoring becomes worth the setup cost. Bringing in that kind of framework on day one is usually premature; not having a path to it by the time you need it is the more common mistake.