arXiv cs.AI
  • Score 83
  • Official

Does the Ranking Hold? A Practitioner-Facing Audit of Hallucination Detectors on Small, Quantized, On-Device LLMs

arXiv:2611.17339v1 Announce Type: new Abstract: Hallucination detection for large language models is dominated by methods evaluated on 7B-70B research models with abundant sampling budgets and exact-match or ROUGE correctness labels. The small models practitioners actually ship on-device, Phi-3-mini, Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct, and Gemma-2-2B-Instruct, typically at 4-bit precision,

So what

What this event means by reading role—not a longer recap.

  • BuilderDon't assume published detector rankings transfer to your 4-bit on-device model; benchmark per model and consider the cheap lexical baseline first.
  • ResearcherDetector rankings can invert across models (Spearman rho -0.80), and the TruthfulQA token-overlap labeler yields near-chance AUROC, so labeler validity needs au

Score dimensions

Higher total means read first. Each bar is one factor we use to rank the system pool. How we score

RelevanceHow tightly this is about AI.
78
ImpactHow much this could change the field or the market.
55
NoveltyHow new this is versus a recap.
70
CredibilityHow much we trust the source.
92
ActionabilityWhether a reader can do something with it.
65