A knowledge assistant is only as trustworthy as its sources, retrieval, answers, and operating controls.
This framework evaluates retrieval-augmented generation and enterprise knowledge systems as a chain. It separates corpus health, retrieval quality, answer grounding, end-to-end task value, and production controls so a polished response cannot hide a weak system.
Evaluate five layers independently, then test the workflow end to end.
Are approved sources complete, current, attributable, permissioned, and suitable for retrieval?
Does the system consistently find the evidence required to answer each question?
Does the answer stay supported by retrieved evidence, preserve nuance, and cite correctly?
Does the complete experience improve accuracy, resolution time, consistency, or another real baseline?
Are permissions, freshness, logging, monitoring, ownership, and escalation reliable in production?
Ship only when critical slices meet explicit thresholds and known failures have safe behavior.
Each failure requires a metric at the layer where it begins.
| Layer | Question | Useful measures | Typical failure |
|---|---|---|---|
| Corpus | Does the answerable evidence exist and remain governed? | Coverage, freshness, duplication, ownership, access accuracy | The right answer is absent, stale, or exposed to the wrong user |
| Retrieval | Did the needed evidence appear in the retrieved set? | Recall at k, precision at k, rank quality, slice-level miss rate | Relevant evidence exists but never reaches the model |
| Generation | Is the response supported, relevant, complete, and clear? | Claim support, citation correctness, citation completeness, answer relevance | The response adds unsupported claims or drops an important condition |
| Behavior | Does the system respond safely when it should not answer? | Abstention precision, refusal quality, conflict handling, injection resistance | The assistant answers confidently when evidence is missing or contradictory |
| Workflow | Does the system improve the human or business task? | Task success, review effort, resolution time, escalation rate, user correction rate | Offline scores improve while the operating workflow gets slower or riskier |
| Operations | Does quality persist under production conditions? | Latency, cost, freshness lag, availability, drift, incident and feedback trends | A strong pilot degrades as sources, users, and query patterns change |
Build tests from real work, not a handful of showcase prompts.
- Sample real questions across roles, sources, workflow stages, and difficulty.
- Record the required evidence, acceptable answer, prohibited claims, and expected citations.
- Include unanswerable, ambiguous, conflicting, stale, permission-sensitive, and adversarial cases.
- Tag each case by business risk so critical failures do not disappear inside an average score.
- Version the set and keep a holdout portion for meaningful regression testing.
A reference answer alone is not enough. Record which evidence must support it and what safe behavior looks like when that evidence is unavailable.
First ask whether the answer had a fair chance to be correct.
For each question, check whether the retrieved set contains every passage required for a sufficient answer.
Measure whether the best evidence appears early enough to influence the response within the context budget.
Test metadata, tenant, role, geography, date, and document-state filters as security and quality controls.
Include acronyms, paraphrases, misspellings, underspecified questions, and terminology used by different teams.
If required evidence was not retrieved, fix indexing, chunking, metadata, query handling, or ranking before tuning the answer prompt.
Score claims against evidence, not against fluency.
- Grounding: each material claim is supported by retrieved evidence
- Citation correctness: each cited source actually supports the adjacent claim
- Citation completeness: consequential claims are cited, not merely the easy ones
- Relevance: the response answers the user's question without distracting material
- Completeness: required conditions, exceptions, and conflicts are preserved
- Abstention: the system declines or escalates when support is insufficient
Automated graders can accelerate testing, but they should be calibrated against human review. High-risk slices need expert judgment and explicit acceptance criteria.
Use thresholds by risk slice, not one universal quality score.
No permission leakage, prohibited action, or unsupported high-consequence instruction in the critical test set.
Retrieval, grounding, citations, and abstention meet documented targets for each important query class.
Users complete the target task at least as accurately and efficiently as the agreed baseline.
Owners, logs, feedback review, source-refresh checks, rollback, and incident escalation are ready before release.
Evaluation continues after launch.
- Log queries, retrieved source identifiers, citations, responses, latency, and system version with appropriate privacy controls
- Route user corrections and escalations into a reviewed failure taxonomy
- Run the regression suite when models, prompts, chunking, ranking, or sources change
- Monitor quality by role, source, query class, and risk level—not only aggregate traffic
The system needs a named owner for source quality, evaluation, user feedback, and release decisions. “The model” cannot own an incident.
Turn a convincing prototype into evidence for a controlled release.
Use the Workflow Suitability Framework first to confirm that RAG is the right pattern. Then apply this evaluation framework before pilot approval, production release, and every material system change.