Finding the Right Knowledge
Intelligently. Compliantly.

Retrieval hardware is fundamentally different from ingestion hardware. Ingestion is optimised for throughput — processing millions of documents over hours. Retrieval is optimised for latency — answering queries in under 200ms. This means dedicated GPU or high-CPU nodes for ANN search, separate Redis cluster nodes for semantic caching, and a distinct inference node for the cross-encoder reranker. The KMS decrypt at query time is the piece most architectures miss: if vector payloads are stored encrypted (required for regulated industries), they must be decrypted in memory at retrieval time, which requires a KMS call on every query. That call adds ~5–15ms and must be budgeted into the p95 latency SLA. Data residency zone assignment also happens here — a query containing patient data may need to be routed to an EU-region endpoint before touching any vector search infrastructure.

Before a query touches any retrieval infrastructure, eight controls must pass in sequence. User identity verification via SSO/session token. RBAC pre-check confirming the user role is permitted to query this namespace at all. Query event log — writing user identity, timestamp, source IP, and SHA-256 hash of the query to an immutable log before retrieval begins, not after. Rate limiting as an availability control. PII pre-screen flagging any query that contains personal data before it enters retrieval. Toxicity filter blocking adversarial prompts designed to extract unauthorised information. Data residency routing directing the query to the correct regional endpoint. The query SHA-256 fingerprint creates a tamper-evident chain from query to retrieved context to LLM response — the SOC 1 auditor traces this chain to verify that what was logged is what was answered.

The vector database is not a single system — it is an architecture of four complementary stores. The primary vector store (Qdrant, Weaviate, Pinecone) handles dense embedding search with native namespace isolation and metadata payload filtering. The sparse index (Elasticsearch, OpenSearch) handles exact-term BM25 retrieval — regulation codes, product identifiers, contract numbers that dense retrieval systematically misses. The graph store (Neo4j) handles entity relationships that neither dense nor sparse retrieval can traverse — "show me all policy documents referenced by this contract, and all clauses in those documents that conflict with our internal guidelines." The semantic cache (Redis/GPTCache) stores recent query-result pairs and returns cached results on near-duplicate queries, eliminating retrieval latency entirely for repeated questions — typically hitting on 20–40% of production queries in enterprise deployments.

Namespace isolation is the most important structural control in the retrieval layer. It is placed between indexing strategy and query understanding deliberately — access boundaries must be enforced before the query is even formulated for retrieval. The isolation is structural: an HR query physically cannot return financial documents because the query is scoped to the HR namespace at the vector DB filter layer, not through application logic that could be bypassed. Sensitivity filters enforce document classification — a user without "restricted" clearance never receives restricted-classified chunks regardless of query phrasing. The per-query access log records which namespaces were queried, by whom, at what time — this is the retrieval-layer equivalent of Zone D in ingestion, and it feeds the SOC 1 Type 2 evidence package.

Query understanding is where most RAG implementations fail silently. A user asks "what was discussed in the Q3 board meeting about the India expansion?" — without query decomposition, this becomes a single vector search that misses most relevant context. With query understanding: intent classification identifies this as a multi-document analytical query. Query expansion adds synonyms and related terms. HyDE generates a hypothetical answer document and uses its embedding to retrieve semantically relevant chunks rather than searching for the literal question. Query decomposition breaks it into sub-queries: "Q3 board meeting documents" and "India expansion strategy discussion." Step-back reformulation abstracts to "strategic expansion discussions 2024" to catch related context. The result: 3–5× improvement in retrieval recall before a single vector search is executed. This is the layer that separates demo-quality RAG from production-grade RAG.

All three tracks fire simultaneously — not sequentially. Sequential retrieval wastes latency proportional to the number of tracks. The dense track (ANN vector search, ColBERT late interaction) handles semantic similarity — "what documents are conceptually related to this query?" The sparse track (BM25, SPLADE) handles exact-match precision — regulation codes, product identifiers, policy numbers, names. A query for "Section 14.2(b) of the Master Services Agreement" will score zero on dense retrieval and perfect on sparse. The metadata filter track enforces temporal and access constraints — current-version documents only, documents owned by the querying department, documents within the user's sensitivity clearance. All three tracks produce candidate pools simultaneously; the fusion stage reconciles them.

Reciprocal Rank Fusion (RRF) merges the three candidate pools into a single ranked list without requiring score calibration between tracks. Dense scores and sparse scores exist on different scales — RRF uses rank position rather than raw score, so a chunk ranked 3rd by dense and 8th by sparse fuses cleanly with a chunk ranked 1st by sparse and 18th by dense. Score normalisation is applied per-track before fusion for any weighted ensemble variants. Deduplication removes chunks that appeared in multiple tracks (same chunk retrieved by both dense and sparse — count it once, weight it higher). The output is a top-20 candidate pool entering the reranker.

Initial retrieval returns semantic similarity — not relevance. A cross-encoder reranking model scores each candidate chunk against the full original query for relevance, not vector proximity. The cross-encoder reads the query and chunk together, producing a precise relevance score that bi-encoder embedding models cannot produce. Cohere Rerank and similar managed rerankers offer API-based reranking without running the model locally. MMR (Maximal Marginal Relevance) diversity scoring eliminates redundant chunks — if three chunks are near-identical policy clauses, only the highest-scoring one enters the context window. The output is a top-5 precision context with a 0–1 confidence score per chunk. This confidence score feeds directly into the compliance gate that follows.

This is the most important compliance control in the entire retrieval layer. If confidence scores are below threshold, the system must decline to answer — not fabricate a plausible-sounding response. The threshold is configurable per use case: a customer support bot might gate at 0.65, a medical protocol system at 0.85, a legal compliance system at 0.90. Below-threshold queries are not failures — they are the system working correctly. They route to a human review queue with the original query, the reason for declination, and the highest confidence score achieved. Every declination is logged immutably: query hash, timestamp, confidence scores, and routing decision. This log is SOC 1 evidence that the system refused to hallucinate. ISO 9001 §8.7 (nonconforming output control) requires exactly this: a documented process for handling outputs that fail quality standards, not silent pass-through.

Context assembly packs the top-K chunks into the LLM context window with three constraints: token budget (the total context must fit the model's context window), citation mapping (every chunk is tagged with its source document, version, and chunk ID so the LLM can cite it), and small-to-big retrieval (the reranker selected small precise chunks; this stage retrieves their parent sections for fuller context). The parent retrieval strategy is particularly important for legal and policy documents — a 200-token clause needs to be read alongside its surrounding section to be interpretable. The citation map is the mechanism that makes hallucination detection possible downstream: if the LLM makes a claim, it must be traceable to a specific chunk ID in this map.

Before assembled context leaves the retrieval layer and enters the LLM, four output controls fire. Output PII filter removes any personally identifiable information from chunks before the LLM processes them — even if a user has clearance to retrieve a document, the LLM should not receive raw PII that it might reproduce in its output. RBAC re-verify confirms at retrieval time that the access permissions established at Zone A still apply — session tokens can expire, permissions can be revoked mid-session. Provenance lock tags every chunk with its exact source document ID, version number, and ingestion timestamp — this chain from ingestion lineage ID through to LLM context is the complete audit trace. The immutable WORM retrieval log records the full query trace: query hash, namespace accessed, chunk IDs retrieved, confidence scores, routing decision, and context sent to LLM. This is the SOC 1 Type 2 evidence package for the retrieval layer — 12 months of logs demonstrating that access controls operated continuously.

Retrieval observability is what separates a deployed system from a monitored one. Prometheus metrics collect query-to-response latency broken down by stage (ANN search time, reranker time, cache hit/miss), confidence score distributions per query category, cost per query (embedding inference, reranker API calls, cache tier cost), and retrieval miss-rate — the rate at which queries fail to retrieve any above-threshold candidates. Miss-rate is the silent failure signal: it tells you that your knowledge corpus has gaps the ingestion pipeline didn't fill. A sustained miss-rate above 5% on a query category signals a corpus gap, not a retrieval failure — and triggers the feedback loop back to ingestion.

The feedback loop is what makes a retrieval system improve over time rather than decay. RAGAS (Retrieval Augmented Generation Assessment) runs offline against a golden dataset, measuring context precision (how much of the retrieved context was actually used), context recall (how much relevant context was missed), and faithfulness (whether LLM claims are grounded in retrieved context). A/B testing compares retrieval strategies — testing whether BM25 hybrid outperforms pure dense on a specific query category, or whether a narrower reranker threshold improves precision at the cost of recall. User signal loop captures explicit feedback (thumbs up/down) and implicit signals (did the user rephrase and ask again?). Index re-tune triggers fire when RAGAS scores drop below baseline or miss-rate rises — initiating a re-indexing run, embedding model evaluation, or chunking strategy review in the ingestion pipeline. This is the feedback arrow that connects retrieval back to ingestion.