From Raw Document
To Governed Answer.

Every document that passes through ingestion enters processing with a lineage ID and a sensitivity label. Metadata enrichment fills in the structured attributes that make retrieval meaningful rather than merely possible. Source tagging records the originating system, document ID, and origin URL — creating the first link in the audit chain that must be traceable all the way to the final LLM response. Version control records the parent-child relationship between document versions: when a policy is updated, v2 knows it superseded v1, enabling the retrieval layer to enforce current-only or historical queries by design. Access tier classification assigns the document to one of four levels (public, internal, confidential, restricted) based on the sensitivity label propagated from ingestion. Effective date enables temporal scoping — a compliance query can be answered with "the policy as it existed on 1 January 2024" rather than just the current version. Department scope assigns the namespace that will govern retrieval isolation. None of this is cosmetic metadata — every attribute is used by the retrieval layer to filter, scope, and audit queries at the infrastructure level.

Content classification assigns a semantic type to each document segment — procedural, regulatory, definitional, advisory, or technical. This classification is not cosmetic: it enables the retrieval layer to filter by content type in addition to semantic similarity and keyword match. A compliance officer asking "what does our policy say about data retention?" needs definitional and regulatory content, not procedural steps. A field technician asking "how do I reset the device?" needs procedural content, not regulatory background. Without content classification, the retrieval layer returns everything semantically relevant — which is often too broad. Classification is applied at the chunk level, not just the document level, because a single document can contain clauses of different types. A master services agreement contains definitional sections (what terms mean), regulatory sections (what laws apply), and procedural sections (what happens when either party breaches). Retrieving all of them for a simple definitional query inflates context noise and degrades LLM response precision.

Before chunks enter the embedding pipeline, four quality controls fire. PII detection classifies any chunk containing personally identifiable information — names, account numbers, health data, financial records — and applies the appropriate sensitivity label. Chunks containing PII are not blocked from processing; they are tagged so that retrieval-layer access controls can enforce who is permitted to receive them. Content quality check enforces a minimum token threshold — chunks below 50 tokens are either merged with adjacent context or discarded with a processing log entry. Sub-threshold chunks are almost always parsing artefacts (page headers, footers, figure captions) that produce low-quality embeddings and dilute retrieval precision. Duplicate check using SimHash (not SHA-256) detects near-duplicate chunks — the same policy clause that appears in two different documents with minor wording differences would otherwise produce two nearly identical embeddings that compete in retrieval, inflating apparent relevance of duplicated content. Processing log records every decision for ISO 9001 §9.1 monitoring and measurement evidence: what was processed, what was flagged, what was discarded, and why.

Fixed-size character chunking is the single most common cause of enterprise RAG failure. Splitting text at 512-character boundaries without regard for sentence structure, clause boundaries, or paragraph semantics produces fragments that are syntactically correct but semantically broken. The standard enterprise approach uses semantic boundary detection — identifying logical breakpoints based on sentence completion, paragraph endings, section headers, and clause structure. Parent-child chunking is the architecture pattern that resolves the precision-context tradeoff: small chunks (200–300 tokens) are indexed for precise retrieval, but each small chunk carries a reference to its parent section (600–1000 tokens) which is retrieved at inference time for fuller context. This means the retrieval layer finds the right precise chunk, and the inference layer receives enough surrounding context to interpret it correctly. Sliding window overlap (default 20%) prevents context loss at chunk boundaries — the last 20% of one chunk is repeated as the first 20% of the next, ensuring that information spanning a boundary appears in at least one complete chunk. The 512-token sweet spot is empirically derived: below 256 tokens, chunks are too short to contain complete semantic units in most enterprise documents; above 1024 tokens, chunks contain multiple distinct topics that degrade retrieval precision.

Embedding model selection is the decision with the highest downstream impact on retrieval quality and the one most frequently made incorrectly. Generic embedding models (trained on web text) systematically misrepresent domain-specific terminology: in a legal corpus, "consideration" means something precise and contractual; in a medical corpus, "discharge" has multiple distinct meanings depending on clinical context. Generic models treat these as their web-frequency meanings. Domain fine-tuning on 5,000–10,000 in-domain query-document pairs produces embeddings that understand regulatory terminology, cross-references, and domain-specific synonyms correctly — consistently delivering 20–30% precision improvement over generic models in enterprise evaluation datasets. The dimensionality decision is a cost-quality tradeoff: 768-dimension models (MiniLM, BGE-small) are 4× cheaper to store and query than 1536-dimension models (text-embedding-3-small) and sufficient for most enterprise Q&A workloads. 3072-dimension models (text-embedding-3-large) are reserved for high-stakes domains where marginal recall improvement has direct monetary or regulatory value. The embedding cache is the most consistently underutilised cost-reduction mechanism in production RAG systems: if a chunk's content has not changed since its last embedding (detectable by SHA-256 content hash comparison), re-embedding on incremental ingestion runs is pure waste. Embedding cache hit rates of 60–80% are typical in enterprise deployments with stable knowledge corpora.

Writing embeddings to the vector database is not a simple insert operation in a production system. Upsert strategy with chunk-level idempotency keys (SHA-256 of chunk content + document ID + version) means that re-running the processing pipeline on a partially updated corpus does not create duplicate embeddings — if a chunk already exists with the same content and version, the write is a no-op. Batch write optimisation groups embedding writes to maximise GPU batch efficiency and minimise per-write overhead — individual writes are 10–50× more expensive than batched writes at scale. Namespace assignment at write time is the structural control that enforces retrieval isolation: a chunk written to the HR namespace cannot be retrieved by a query scoped to the finance namespace, regardless of semantic similarity. Index versioning tags the write with the current index version, enabling blue-green deployment of index updates (the new index version serves queries while the old version remains as rollback), A/B testing of embedding model changes, and point-in-time recovery of the index state as it existed before a large corpus update.

The HNSW (Hierarchical Navigable Small World) index that powers ANN retrieval degrades in quality as the corpus changes significantly — chunks are added, updated, and deleted, but the graph structure of the index was optimised for the original corpus distribution. HNSW rebuild triggers fire when cumulative corpus change exceeds a configurable threshold (typically 10–15% of total chunk count) or on a scheduled basis for high-churn corpora. Staleness detection monitors embedding age against document update frequency — if a knowledge domain is updated regularly but re-indexing has not occurred, retrieved chunks may represent stale knowledge that does not reflect current policy. Tiered storage assigns chunks to hot (GPU-resident, <5ms retrieval), warm (SSD-backed, 10–30ms), and cold (object storage, 100–500ms) tiers based on query frequency — recent regulatory documents are hot, archived historical versions are cold. Delete propagation is the most operationally critical index management function: when a document is deleted from the source system or expired by retention policy, every chunk derived from that document must be purged from the vector index, the sparse index, and the graph store simultaneously. Partial deletion creates phantom chunks — orphaned embeddings that remain retrievable after the source document has been deleted, creating both compliance violations and retrieval quality degradation.

Prompt architecture is an engineering discipline, not a craft activity. System prompts that are written once, never tested, and never versioned are the primary cause of production LLM failures in enterprise deployments — not model capability limitations. Every system prompt must be version-controlled (semantic versioning: 1.0.0 → 1.0.1 for wording changes, 1.1.0 for structural changes, 2.0.0 for fundamental redesign), tested against a golden dataset before deployment, and locked to a specific hash in production to prevent accidental or unauthorised modification. Context injection is the structured assembly of retrieved chunks into the prompt: source attribution is embedded at the chunk level (not appended as a list at the end), so the LLM can reference specific sources inline. Citation mandate is a hard constraint in the system prompt: the LLM is instructed that every factual claim must be followed by a source reference, and that any claim for which a source cannot be found must not be made — the refusal protocol takes over instead. Refusal protocol defines the exact language and structure of graceful declination: "I was unable to find reliable source material for this question in the available knowledge base. The query has been logged for review." This is logged, not discarded — the accumulation of declined queries is a corpus gap signal that feeds back into ingestion.

LangChain provides the component chaining framework — retrieval, reranking, prompt assembly, and response generation are connected as a pipeline with defined interfaces between each stage. LangGraph extends this with stateful multi-step reasoning: conditional routing (if the initial retrieval confidence is below threshold, reroute to a broader retrieval strategy before escalating), loops (plan → retrieve → evaluate → retrieve again if insufficient), and human-in-the-loop interrupts (pause the workflow at a defined checkpoint to await human review before proceeding). Agent routing dispatches complex multi-part queries to specialist sub-agents: a query that requires both regulatory lookup and financial calculation is decomposed into a regulatory retrieval agent and a calculation tool agent, whose outputs are synthesised by an orchestrator agent. Tool calling enables the LLM to invoke external capabilities mid-reasoning: a structured database query for exact figures, a calculator for numerical operations, a web search for information beyond the knowledge base. Memory systems maintain session-level short-term memory (what has been asked and answered in this conversation) and cross-session long-term memory (user preferences, recurring query patterns, personalisation signals) — both with appropriate data retention constraints for regulated environments.

Model selection at inference time is a cost-quality routing decision that most production systems get wrong by defaulting to the most capable model for every query. A factual lookup query ("what is the maximum file size for uploaded documents?") requires a small, fast, cheap model — the complexity of GPT-4 adds latency and cost without improving the answer. A multi-document synthesis query ("summarise the key obligations under all active vendor contracts and identify any conflicts") requires the most capable model available. Cost routing maintains a complexity classifier that assigns each query to one of three tiers: simple factual (small model, <50ms, ~$0.001), moderate analytical (medium model, <150ms, ~$0.01), complex synthesis (large model, <500ms, ~$0.10). On-premise deployment with quantized models (Llama 3 8B at 4-bit quantization, Mistral 7B) is mandatory for industries where data sovereignty prohibits sending content to external API endpoints — financial data, clinical records, legal documents. Quantized models run at 60–70% of full-precision quality at 15–20% of the compute cost, which is an acceptable tradeoff for most enterprise Q&A workloads. Fallback models activate when the primary model is unavailable, rate-limited, or returns an error — the fallback chain must be defined, tested, and logged so that a model outage does not produce user-visible failures.

Before a response is delivered to the user, four output controls fire against the generated text. Citation verification checks that every factual claim in the response is traceable to a specific chunk ID in the context window — this is the primary hallucination detection mechanism, not an NLI model. If a claim appears in the response but cannot be matched to a source chunk, the claim is either flagged for review or the entire response is suppressed and replaced with a graceful declination. NLI-based hallucination signal uses a Natural Language Inference model to score each claim against its cited source chunk for entailment (the claim is supported by the source), contradiction (the claim contradicts the source), or neutrality (the claim cannot be verified from the source). High contradiction scores trigger immediate response suppression; high neutrality scores flag the response for human review. Output PII filter scans the generated response for personally identifiable information that should not have been included — a last-resort check before delivery. Format and length check enforces output constraints defined in the system prompt: structured JSON responses must validate against the defined schema, prose responses must fall within the defined length range, and domain-restricted responses must not contain content outside the permitted scope.

Response generation produces the final structured output that is delivered to the user. Streaming responses (token-by-token generation) are appropriate for conversational interfaces where perceived latency matters more than total latency. Batch responses are appropriate for background processing workflows where throughput matters more than perceived latency. Structured output enforces JSON schema validation when the application requires machine-readable responses rather than prose — a compliance reporting tool that generates audit summaries must produce valid, schema-conforming JSON that downstream systems can consume without parsing. Confidence score annotation attaches the retrieval confidence score and the citation verification status to the response metadata — not displayed to end users in most interfaces, but available to application developers to drive UI decisions (show a "verify this answer" warning below a confidence threshold, for example). Source attribution is the mechanism that makes the system auditable: every response includes a citation list linking each factual claim to the specific chunk ID, source document, document version, and effective date that supports it. This citation chain is what allows a compliance officer to reconstruct exactly what information the system used to answer a specific query at a specific point in time.

Post-generation constraints are the last safety layer before response delivery. Toxicity check scans the response for harmful content — particularly important when the LLM processes documents that may contain adversarial content injected into the knowledge base, a technique known as indirect prompt injection. Scope boundary enforcement verifies that the response stays within the domain the system is permitted to answer — a legal compliance assistant should not be answering questions about competitors' products, even if the retrieval layer happened to surface a document containing that information. Human-in-the-loop routing applies to queries designated as high-stakes: medical dosing calculations, legal advice that will be acted upon without further review, financial decisions above a defined threshold. These queries are intercepted before delivery and queued for human review — the AI-generated response is a draft, not a final answer. Escalation queue maintains SLA commitments for human review — if a query sits in the queue for more than the defined SLA period without review, an alert fires and the query is escalated to a supervisor. The SLA must be documented and auditable for SOC 1 Type 2 evidence.

The response audit trail is the complete chain of evidence from query to answer that a compliance officer, internal auditor, or external regulator can use to reconstruct exactly what the system did, when it did it, with what information, using which model, under which prompt version. Full response log writes the complete record to WORM (Write Once Read Many) immutable storage: the original query text, the retrieved chunk IDs with their confidence scores, the assembled prompt (including system prompt version hash), the model identifier and configuration (temperature, max tokens, top-p), the raw model response, the citation verification result, and the final delivered response. This log cannot be modified after writing — it is the forensic record. Prompt version log records which system prompt version was active at the time of each query — when a prompt is updated, the previous version is archived, not deleted, so that any past query can be replayed with the exact prompt it was answered with. Model configuration log records model ID, quantization level, and inference parameters — because the same model at different temperatures produces systematically different outputs, and the log must capture the exact configuration. The WORM audit store satisfies SOC 1 Type 2's requirement for evidence that controls operated effectively over a 12-month period — the immutability of the log is what makes that evidence credible to an auditor.

Cost tracking per query is the operational control that prevents AI infrastructure costs from becoming invisible until the invoice arrives. Token monitoring counts input tokens (context window + prompt) and output tokens (generated response) for every query, attributed to the department, use case, and user tier that initiated it. Cost per query is computed in real time and attributed to the requesting department — this is the mechanism that enables showback (showing departments their AI infrastructure consumption) and chargeback (billing departments for their actual consumption). Time to first token (TTFT) and tokens per second (TPS) are the two latency metrics that matter for perceived performance: TTFT determines how quickly the user sees the first word of the response (target: <500ms), TPS determines how fast the response streams after it starts (target: >30 tokens/second for fluent reading pace). User feedback signal captures explicit thumbs up/down ratings and implicit signals (query rephrasing, session abandonment, correction submissions) that are the earliest indicators of model drift — when a model that was performing well begins returning lower-quality answers, user feedback signals typically degrade weeks before automated quality metrics reflect the change.