评估LLM在辅导培训中的角色一致性
Evaluating Role-Consistency in LLMs for Counselor Training
AI总结：
本文通过引入对抗性数据集评估LLM在辅导培训中的角色一致性，比较了Vicuna与其他开源模型的表现。
摘要：
在线咨询服务的兴起凸显了未来辅导人员有效培训方法的必要性。本文扩展了VirCo（虚拟客户用于在线辅导）的研究，这是一种旨在补充学术培训中传统角色扮演方法的虚拟客户，通过模拟真实的客户互动。基于先前工作，我们引入了一个包含对抗性攻击的数据集，以测试大型语言模型（LLMs）维持其指定角色（角色一致性）的能力。本研究重点评估Vicuna模型响应的角色一致性和连贯性，并将其发现与早期研究进行比较。此外，我们评估并比较了各种开源LLM在虚拟客户互动中维持角色一致性方面的表现。我们的贡献包括创建对抗性数据集、评估对话连贯性和人格一致性，以及提供不同LLM的比较分析。
英文摘要：
The rise of online counseling services has highlighted the need for effective training methods for future counselors. This paper extends research on VirCo, a Virtual Client for Online Counseling, designed to complement traditional role-playing methods in academic training by simulating realistic client interactions. Building on previous work, we introduce a new dataset incorporating adversarial attacks to test the ability of large language models (LLMs) to maintain their assigned roles (role-consistency). The study focuses on evaluating the role consistency and coherence of the Vicuna model's responses, comparing these findings with earlier research. Additionally, we assess and compare various open-source LLMs for their performance in sustaining role consistency during virtual client interactions. Our contributions include creating an adversarial dataset, evaluating conversation coherence and persona consistency, and providing a comparative analysis of different LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
