CQA-Eval: 设计在资源受限情况下多段临床问答系统可靠评估
CQA-Eval: Designing Reliable Evaluations of Multi-paragraph Clinical QA under Resource Constraints
AI总结：
本文提出CQA-Eval框架，通过医生标注300个真实患者问题，比较粗粒度答案级与细粒度句子级评估在正确性、相关性和风险披露维度上的效果，发现细粒度标注提升正确性一致性，粗粒度提升相关性一致性，而风险披露判断仍不一致，且少量标注可提供与粗标注相当的可靠性。
摘要：
评估多段临床问答系统是资源密集且具有挑战性的任务：准确判断需要医学专业知识，且在多段文本上达成一致的人类判断困难。我们引入CQA-Eval，一种适用于资源受限和高专业性的评估框架及评估建议。基于医生对300个真实患者问题由医生和LLM回答的标注，我们比较粗粒度答案级与细粒度句子级评估在正确性、相关性和风险披露维度上的效果。我们发现，标注者间一致性（IAA）因维度而异：细粒度标注在正确性上提升一致性，粗粒度标注在相关性上提升一致性，而风险披露判断仍不一致。此外，仅标注少量句子可提供与粗标注相当的可靠性，从而降低成本和努力。
英文摘要：
Evaluating multi-paragraph clinical question answering (QA) systems is resource-intensive and challenging: accurate judgments require medical expertise and achieving consistent human judgments over multi-paragraph text is difficult. We introduce CQA-Eval, an evaluation framework and set of evaluation recommendations for limited-resource and high-expertise settings. Based on physician annotations of 300 real patient questions answered by physicians and LLMs, we compare coarse answer-level versus fine-grained sentence-level evaluation over the dimensions of correctness, relevance, and risk disclosure. We find that inter-annotator agreement (IAA) varies by dimension: fine-grained annotation improves agreement on correctness, coarse improves agreement on relevance, and judgments on communicates-risks remain inconsistent. Additionally, annotating only a small subset of sentences can provide reliability comparable to coarse annotations, reducing cost and effort.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
