arXiv 2507.029502026-08-28cs.CLcs.AIcs.HC

固定合成日语咨询对话中的结构化提示与自动评估

Distinct Profiles of Run-to-Run Score Reliability and Expert-Panel Alignment Across Four LLM Evaluators of Simulated Japanese-Language AI-to-AI Counseling

Japan National Institute of Occupational Safety and Health（日本国立职业安全卫生研究所）；Kaze To Taiyo（凯泽・太阳）；Saga Occupational Health Association（Saga职业健康协会）；Department of Pharmacy, Zikei Hospital/Zikei Institute of Psychiatry（药剂科，Zikei医院/Zikei精神医学研究所）；Department of Medical Welfare, Suzuka University of Medical Science（医疗福祉科， Suzuka医科大学）；Graduate School of Human Sciences, Ritsumeikan University（人类科学研究生院，立命馆大学）；Faculty of Nursing, National Defense Medical College（护理学部，国家防卫医疗大学）；Support Center for Students with Disabilities, Aoyama Gakuin University（残疾学生支持中心，上智大学）

Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Gotō, Tomonori Hosokawa, Makoto Nishimura, Yōsuke Satō, Izumi Sezai, Tomohiro Inoue

备注：: 55 pages, 2 figures, 31 tables; supplemental material included; data and code at https://doi.org/10.5281/zenodo.22106028; preregistration and amendments at https://doi.org/10.17605/OSF.IO/VU286

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究通过人工智能生成的18份固定日语咨询对话记录，对比GPT - minimal、GPT - SMDP和Claude - SMDP三种情况，咨询专家与新的语言模型分别评级，发现SMDP对话在多方面获更高专家评级，语言模型评级可重复但偏宽松。

摘要：

大语言模型可能支持咨询培训，但日语交互和自动质量评级的证据有限。我们研究了在三种咨询条件下通过人工智能到人工智能交互生成的18份固定日语咨询记录。15位咨询专家和三个新的语言模型分别对记录进行评级。结果表明SMDP条件对话在多方面获更高专家评级，语言模型评级可重复但偏宽松。这些发现为日语人工智能咨询模拟提供了专家参考基准。

英文摘要：

Large language models (LLMs) increasingly evaluate generated dialogue, but repeatable scores do not necessarily align with professional judgment. This observational fixed-benchmark study compared four configured LLM evaluator systems (GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8, and Fable 5) with aggregated ratings from 15 counseling experts on 18 complete simulated AI-to-AI counseling sessions conducted in Japanese. The sessions represented three counselor conditions across six prespecified client profiles. Each system scored every transcript three times on four motivational interviewing-informed dimensions and overall quality. All four systems assigned higher scores than the expert panel for softening sustain talk and overall quality, although differences varied across systems and constructs. Single-run intraclass correlation coefficients ranged from .33 to .96, showing that high run-to-run reliability did not ensure closer expert-panel alignment. Claude Opus 4.8 had the smallest mean absolute difference, whereas Fable 5 had an intermediate difference. In a secondary benchmark analysis, GPT-4-turbo sessions generated with the Structured Multi-step Dialogue Prompt received higher expert ratings than sessions generated by the same model with a minimal instruction for cultivating change talk, partnership, empathy, and overall quality; the softening sustain talk contrast remained uncertain. The fixed benchmark contained one session per counselor-condition-by-profile cell, so inference concerns these sessions rather than all possible stochastic regenerations. Run-to-run reliability, expert-panel alignment, and condition discrimination are separate properties of automated counseling evaluation. This benchmark supports construct-level assessment of LLM evaluators against professional judgment.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。