arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00870cs.CLcs.HC
收藏

Roleplay-doh:通过引导并遵循原则,让领域专家能够创建LLM模拟的患者

Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles

Stanford University(斯坦福大学)

Ryan Louie, Ananjan Nandi, William Fang, Cheng Chang, Emma Brunskill, Diyi Yang

AI总结:

针对敏感场景LLM角色扮演的数据限制与专家反馈收集难的问题,研究提出Roleplay-doh协作流程,将专家反馈转化为原则指导角色扮演,新增的原则遵循提示流程提升30%效果,经25名专家验证可高效创建更真实的AI模拟患者。

摘要:

近期研究利用大语言模型(LLMs)扮演真实社交场景,帮助新手练习社交技能。然而,模拟心理健康这类敏感场景的互动颇具挑战:隐私顾虑限制了数据访问,而收集专家反馈虽至关重要却十分耗费人力。为解决这一问题,我们开发了Roleplay-doh——一种新型的人类-LLM协作流程,它从领域专家处获取定性反馈,并将其转化为一套原则(即自然语言规则),用于指导LLM提示驱动的角色扮演。我们应用该流程,让资深心理健康支持者能够创建定制化的AI患者,作为新手咨询师的模拟练习伙伴。在发现GPT-4的模拟存在不遵循专家定义原则的问题后,我们还提出了一种新型的原则遵循提示流程,在下游任务中,其回复质量和原则遵循度提升了30%。通过一项有25名咨询专家参与的用户研究我们证实,该流程能让创建AI患者的过程既简便又有效,且无论是创建者还是第三方咨询师都认为,这些AI患者更贴近真实患者。代码和数据可访问项目网站https://roleplay-doh.github.io/获取。

英文摘要:

Recent works leverage LLMs to roleplay realistic social scenarios, aiding novices in practicing their social skills. However, simulating sensitive interactions, such as in mental health, is challenging. Privacy concerns restrict data access, and collecting expert feedback, although vital, is laborious. To address this, we develop Roleplay-doh, a novel human-LLM collaboration pipeline that elicits qualitative feedback from a domain-expert, which is transformed into a set of principles, or natural language rules, that govern an LLM-prompted roleplay. We apply this pipeline to enable senior mental health supporters to create customized AI patients for simulated practice partners for novice counselors. After uncovering issues in GPT-4 simulations not adhering to expert-defined principles, we also introduce a novel principle-adherence prompting pipeline which shows 30% improvements in response quality and principle following for the downstream task. Via a user study with 25 counseling experts, we demonstrate that the pipeline makes it easy and effective to create AI patients that more faithfully resemble real patients, as judged by creators and third-party counselors. See our project website at https://roleplay-doh.github.io/ for code and data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑