arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.09407cs.CL
收藏

CARE-Bench:一个由专家原则指导的多样化来访者模拟基准,用于评估心理咨询场景中的大语言模型

CARE-Bench: A Benchmark of Diverse Client Simulations Guided by Expert Principles for Evaluating LLMs in Psychological Counseling

Bichen Wang, Yixin Sun, Junzhe Wang, Hao Yang, Xing Fu, Yanyan Zhao, Si Wei, Shijin Wang, Bing Qin

AI总结:

针对现有心理咨询LLM评估基准的不足,本文提出动态交互式基准CARE-Bench,依托专家指导的多样化来访者模拟与多维度心理量表评估模型能力,分析现有模型局限并指明优化方向。

摘要:

心理咨询需求不断增长与服务供给有限之间的错配,推动了大语言模型(LLMs)在该领域的应用研究,因此亟需可靠统一的基准来评估各类LLM的咨询能力。但现有工作存在来访者模拟不专业、静态问答评估形式、评价维度单一的局限,难以有效评估模型应对多样复杂来访者的综合能力。为填补这一空白,我们推出CARE-Bench,这是一个动态交互式自动化基准,基于真实咨询案例提取的多样化来访者档案,依照专家指南完成模拟搭建,依托成熟心理学量表提供多维度性能评估。我们使用CARE-Bench评估了多款通用LLM和专业咨询模型,揭示了它们当前存在的局限。我们还与心理学家合作,详细分析了LLM与不同类型来访者交互时失效的原因,为开发更全面、通用、有效的咨询模型指明了方向。

英文摘要:

The mismatch between the growing demand for psychological counseling and the limited availability of services has motivated research into the application of Large Language Models (LLMs) in this domain. Consequently, there is a need for a robust and unified benchmark to assess the counseling competence of various LLMs. Existing works, however, are limited by unprofessional client simulation, static question-and-answer evaluation formats, and unidimensional metrics. These limitations hinder their effectiveness in assessing a model's comprehensive ability to handle diverse and complex clients. To address this gap, we introduce \textbf{CARE-Bench}, a dynamic and interactive automated benchmark. It is built upon diverse client profiles derived from real-world counseling cases and simulated according to expert guidelines. CARE-Bench provides a multidimensional performance evaluation grounded in established psychological scales. Using CARE-Bench, we evaluate several general-purpose LLMs and specialized counseling models, revealing their current limitations. In collaboration with psychologists, we conduct a detailed analysis of the reasons for LLMs' failures when interacting with clients of different types, which provides directions for developing more comprehensive, universal, and effective counseling models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑