VeriGround: catching RAG hallucinations claim by claim (the paper)

Status: first-author academic paper in IEEE format (Sem V Technical Skills & Research Writing, Panimalar Engineering College) — not yet published. The benchmark, framework, and dashboard are open source. Co-authors: Vignesh R, Sabarish J, Shafeeq S, Rohith V K, Srikesh R.

Retrieval-augmented generation (RAG) conditions an LLM's answer on retrieved documents — but grounding the model in evidence doesn't make it honest. A generated answer can still go beyond the evidence, distort it, or contradict it outright, even when the retriever did its job perfectly. And when the whole response gets one aggregate quality score, nobody can tell you which sentence is the problem.

VeriGround treats claim-level verification as a first-class part of the RAG pipeline: decompose the response into atomic claims, check each claim against the exact evidence chunks the generator was shown, and surface a verdict for every claim while the answer is still streaming.

The three-channel design

Each claim is scored against each evidence chunk along three complementary channels — individually weak, jointly discriminative:

1. Dense semantic similarity — cosine similarity between sentence-transformer embeddings of the claim and the chunk.
2. Entity overlap — the fraction of the claim's named entities (parties, dates, jurisdictions, monetary figures) that also appear in the chunk. Cheap, and it catches claims that introduce an entity that isn't in the evidence at all, even when the prose around it is semantically close.
3. Natural language inference (NLI) — a directional entailment score from a transformer NLI model: P(entail) − P(contradict).

These fuse into a single weighted score per claim-evidence pair. The claim is then judged against its best-matching chunk (the anchor chunk):

F(claim, chunk) = 0.35·similarity + 0.20·entity_overlap + 0.45·NLI

Weights and thresholds (H = 0.72, L = 0.40) were grid-searched on a held-out split, then frozen before evaluation.

Four verdicts — and the override that matters

The fused score maps to one of four verdicts:

• Supported — score above the high threshold
• Partially Supported — between the thresholds (evidence backs part of the claim and is silent or weak on the rest)
• Unsupported — below the low threshold with non-negative NLI
• Contradicted — the NLI signal is strongly negative, which overrides the other rules whenever it fires

That contradiction override is the design's sharpest edge: a contradicting chunk can be lexically close to the claim it refutes, so a similarity-only or plain-threshold scheme would wrongly credit it. Separating "missing evidence" from "actively refuted" is exactly what single-signal systems can't do.

Every verdict gets a human-readable explanation, generated by a lightweight LLM that is constrained to ground its wording in the anchor chunk — quoting or paraphrasing only the specific span that drove the verdict — plus a provenance trail from claim → anchor chunk → source document.

A five-module pipeline

1. Ingest — accepts the query and response as they're produced, so verification starts before generation finishes
2. Extraction — splits claims at sentence boundaries plus a coordination-splitting pass that divides compound sentences joined by conjunctions (a single sentence frequently asserts two claims of different veracity)
3. Encoding & evidence fetch — reads the response's original evidence from the vector index (FAISS / ChromaDB). VeriGround never re-queries the index — it verifies what the model actually saw, so claims are neither penalized for evidence the retriever missed nor credited for evidence the generator never had
4. Entity-overlap pre-filter — drops irrelevant claim-chunk pairs before the expensive NLI pass ever runs
5. Verdict engine — applies the four rules + contradiction override

All five modules talk through a lightweight internal message schema, and the fusion weights/thresholds are externalized as configuration — retuning for a new domain (say, legal text with denser entity structure) is a re-tuning exercise, not a re-engineering one. A live dashboard underlines or shades each claim as verdicts stream in, shows the anchor chunk and per-channel scores on hover, and rolls the response up into a single aggregate integrity score.

Benchmark & results

We built a hand-annotated benchmark of 420 claims from six RAG pipelines — three open-domain QA and three legal/policy-document — labeled by two independent annotators into the four verdict classes, with a third adjudicating disagreements. Weights were tuned on a 60-claim split; the rest was reported untouched.

Across the 420-claim benchmark, VeriGround beats both baselines on every axis we measured — NLI-only in parentheses first, then RAGChecker-style, then VeriGround:

• Verdict accuracy (4-class): 0.74 → 0.80 → 0.86
• Supported F1: 0.83 → 0.87 → 0.91
• Partially Supported F1: 0.64 → 0.71 → 0.78
• Unsupported F1: 0.74 → 0.79 → 0.85
• Contradicted F1: 0.77 → 0.81 → 0.88
• Mean latency / claim: 0.61 s → 0.94 s → 0.52 s
• Explanation usefulness (1–5): 2.9 → 3.4 → 4.2

VeriGround beats both baselines on every axis. The biggest relative gains are exactly where single-signal systems struggle: Partially Supported (the class that needs similarity and entailment together) and Contradicted (the class the NLI override exists for). It's also fastest, because the cheap entity-overlap channel prunes pairs before NLI runs — more signals, lower latency.

Ablation confirms no channel is optional: removing NLI causes the largest accuracy drop (entailment is the strongest signal); removing entity overlap hurts Contradicted/Unsupported most (claims similar to chunks that simply omit the asserted entity); removing similarity has the smallest overall effect but visibly damages Partially Supported F1 (similarity is what matches a claim to its anchor chunk when overlap is sparse).

Main residual error: claim extraction. Coordination-splitting occasionally keeps two differently-verdict claims as one (the source of most Partially Supported errors), and the NLI model under-confidently scores claims that need evidence aggregated across two chunks — logged as a distinct multi-hop failure category.

What's next

• More robust extraction of coordinated and nested claims
• Multi-hop evidence aggregation (support across several chunks, not one anchor)
• Lightweight on-device NLI for edge deployment
• Multilingual validation
• A user study in a live legal-tech deployment
• Learned per-domain weights instead of one global weight vector
• Releasing the 420-claim benchmark and annotation guidelines

Repo + live auditing dashboard: github.com/SabarishR08/VeriGround