TN-Eval：衡量行为治疗笔记质量的评分标准与评估协议
TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes
AI总结：
本研究提出TN-Eval评分标准与评估协议，用于衡量行为治疗笔记质量，发现LLM生成的笔记在盲测中被治疗师认为优于人工笔记，但存在幻觉问题。
摘要：
行为治疗笔记对于法律合规和患者护理都至关重要。与身体健康领域的进展记录不同，行为治疗笔记的质量标准仍不完善。为弥补这一空白，我们与持证治疗师合作，设计了一套全面的评分标准，用于在关键维度上评估治疗笔记：完整性、简洁性和忠实性。此外，我们扩展了一个公开的行为健康对话数据集，加入了治疗师撰写的笔记和LLM生成的笔记，并应用我们的评估框架来衡量它们的质量。我们发现：（1）基于评分标准的人工评估协议比传统的李克特量表标注提供更可靠和可解释的结果。（2）LLM在评估完整性和简洁性方面可以模仿人类评估者，但在忠实性方面存在困难。（3）治疗师撰写的笔记往往缺乏完整性和简洁性，而LLM生成的笔记包含幻觉。令人惊讶的是，在盲测中，治疗师更喜欢并认为LLM生成的笔记优于治疗师撰写的笔记。
英文摘要：
Behavioral therapy notes are important for both legal compliance and patient care. Unlike progress notes in physical health, quality standards for behavioral therapy notes remain underdeveloped. To address this gap, we collaborated with licensed therapists to design a comprehensive rubric for evaluating therapy notes across key dimensions: completeness, conciseness, and faithfulness. Further, we extend a public dataset of behavioral health conversations with therapist-written notes and LLM-generated notes, and apply our evaluation framework to measure their quality. We find that: (1) A rubric-based manual evaluation protocol offers more reliable and interpretable results than traditional Likert-scale annotations. (2) LLMs can mimic human evaluators in assessing completeness and conciseness but struggle with faithfulness. (3) Therapist-written notes often lack completeness and conciseness, while LLM-generated notes contain hallucination. Surprisingly, in a blind test, therapists prefer and judge LLM-generated notes to be superior to therapist-written notes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
