- Score 83
- Official
Does the Ranking Hold? A Practitioner-Facing Audit of Hallucination Detectors on Small, Quantized, On-Device LLMs
arXiv:2611.17339v1 Announce Type: new Abstract: Hallucination detection for large language models is dominated by methods evaluated on 7B-70B research models with abundant sampling budgets and exact-match or ROUGE correctness labels. The small models practitioners actually ship on-device, Phi-3-mini, Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct, and Gemma-2-2B-Instruct, typically at 4-bit precision,
So what
What this event means by reading role—not a longer recap.
- BuilderDon't assume published detector rankings transfer to your 4-bit on-device model; benchmark per model and consider the cheap lexical baseline first.
- ResearcherDetector rankings can invert across models (Spearman rho -0.80), and the TruthfulQA token-overlap labeler yields near-chance AUROC, so labeler validity needs au
Score dimensions
Higher total means read first. Each bar is one factor we use to rank the system pool. How we score