Know2Guess: 一种面向大型语言模型知识边界评估的污染感知多区域基准
Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models
AI总结：
提出一种污染感知多区域基准，通过冻结构建时标签测量从可回答知识到应弃权未知的转变，评估模型在回答与弃权间的可靠性。
摘要：
大型语言模型的可靠评估应将支持性回答与无根据猜测区分开，同时避免与数据污染、提示特异性或通用拒绝行为混淆。我们提出一种污染感知的多区域基准，用于在冻结构建时标签下测量从可回答知识到应弃权未知的转变。该基准包含五个领域的1200个项目、明确的弃权预期、污染风险元数据，以及双重解析（官方严格解析器加归一化鲁棒解析器）。我们在锁定回答或弃权提示、仅回答控制以及提示模板变体下评估了FLAN-T5、Qwen2.5-Instruct和Llama-3-Instruct模型。该基准未被通用非回答行为解决：FLAN基线在有效弃权方面仍然薄弱，而更强的指令调优模型暴露出从回答到弃权的选择性但不完全的转变。Qwen2.5-3B-Instruct实现了最佳整体可靠性，但预期回答区域仍然困难，校准仍然较差，良性项目拒绝仍然存在。提示和解析器鲁棒性分析保留了主要排名和定性结论。因此，该基准为审计可回答性、弃权、拒绝和污染作为LLM评估中不同但相互作用的维度提供了可重复的协议。数据集公开于https://this https URL。
英文摘要：
Reliable evaluation of large language models should separate supported answering from unsupported guessing without conflating either with data contamination, prompt idiosyncrasy, or generic refusal behavior. We present a contamination-aware, multi-zone benchmark for measuring the transition from answerable knowledge to abstention-expected unknowns under frozen build-time labels. The benchmark contains 1,200 items across five domains, explicit abstention expectations, contamination-risk metadata, and dual parsing with an official strict parser plus a normalized robustness parser. We evaluate FLAN-T5, Qwen2.5-Instruct, and Llama-3-Instruct models under locked answer-or-abstain prompts, answer-only controls, and prompt-template variants. The benchmark is not solved by generic non-answer behavior: FLAN baselines remain weak on productive abstention, while stronger instruction-tuned models expose a selective but incomplete transition from answering to abstaining. Qwen2.5-3B-Instruct achieves the best overall reliability, but answer-expected zones remain difficult, calibration remains poor, and benign-item refusal persists. Prompt and parser robustness analyses preserve the main ranking and qualitative conclusions. The benchmark therefore provides a reproducible protocol for auditing answerability, abstention, refusal, and contamination as distinct but interacting dimensions of LLM reliability.The dataset is publicly available at https://github.com/renweimeng/Know2Guess-A-Contamination-Aware-Multi-Zone-Benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
