The mismatch between the growing demand for psychological counseling and the limited availability of services has motivated research into the application of Large Language Models (LLMs) in this domain. Consequently, there is a need for a robust and unified benchmark to assess the counseling competence of various LLMs. Existing works, however, are limited by unprofessional client simulation, static question-and-answer evaluation formats, and unidimensional metrics. These limitations hinder their effectiveness in assessing a model's comprehensive ability to handle diverse and complex clients. To address this gap, we introduce \textbf{CARE-Bench}, a dynamic and interactive automated benchmark. It is built upon diverse client profiles derived from real-world counseling cases and simulated according to expert guidelines. CARE-Bench provides a multidimensional performance evaluation grounded in established psychological scales. Using CARE-Bench, we evaluate several general-purpose LLMs and specialized counseling models, revealing their current limitations. In collaboration with psychologists, we conduct a detailed analysis of the reasons for LLMs' failures when interacting with clients of different types, which provides directions for developing more comprehensive, universal, and effective counseling models.
PsyDT: Using LLMs to Construct the Digital Twin of Psychological Counselor with Personalized Counseling Style for Psychological Counseling
School of Future Technology, South China University of Technology(华南理工大学未来技术学院);School of Electronic and Information Engineering, South China University of Technology(华南理工大学电子与信息工程学院);Pazhou Lab(琶洲实验室)
AutoCBT: An Autonomous Multi-agent Framework for Cognitive Behavioral Therapy in Psychological Counseling
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);University of Macau(澳门大学);Shenzhen University(深圳大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))
KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors
The University of Electro-Communications(电通通讯大学);Rapport Technologies, Inc.(Rapport技术公司);iDEAR Human Support Service(iDEAR人类支持服务);Japanese Organization of Mental Health and Educational Agencies(日本心理健康与教育机构联合会)
UKP Lab, Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt(德国达姆施塔特技术大学UKP实验室、计算机科学系和海森人工智能中心(hessian.AI));Zuse School ELIZA(Zuse学院ELIZA);National Research Center for Applied Cybersecurity ATHENE(应用网络安全国家研究中心ATHENE);Indian Institute of Technology Delhi(印度德里理工学院);Yardi School of Artificial Intelligence(Yardi人工智能学院);University of Louisville(路易斯维尔大学);University of Toledo(托莱多大学)
CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling
Huazhong University of Science and Technology(华中科技大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Macau(澳门大学);University of Science and Technology of China(中国科学技术大学);Jilin University(吉林大学);South China University of Technology(华南理工大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学)
Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback
Central South University(中南大学);CoAI Group(CoAI小组);DCST(国防科学技术大学);IAI(人工智能院);BNRIST(北京航空航天大学);Tsinghua University(清华大学);Lingxin AI(灵犀AI);Fuxi AI Lab(伏羲AI实验室);NetEase Inc.(网易公司)
MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessions
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt(德累斯顿技术大学计算机科学系、普遍知识处理实验室(UKP Lab)、黑森人工智能中心(hessian.AI));National Research Center for Applied Cybersecurity ATHENE, Germany(应用网络安全国家研究中心ATHENE,德国);Department of Electrical Engineering, Indian Institute of Technology Delhi, India(印度德里印度理工学院电气工程系);Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, India(印度德里印度理工学院人工智能学院)
Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济发展实验室);SKL-IOTSC, CIS, University of Macau(澳门科学馆物联网与智慧城市研究中心);Shandong University(山东大学);Auckland University of Technology(技术大学(奥克兰));University of Electronic Science and Technology of China(电子科技大学)
Enhancing Psychological Counseling with Large Language Model: A Multifaceted Decision-Support System for Non-Professionals
Sorbonne Université(索邦大学);Institut du Cerveau - Paris Brain Institute - ICM(巴黎脑科学院);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);Inserm(法国国家健康与医学研究院);AP-HP(巴黎公立医院集团);Hôpital de la Pitié Salpêtrière(萨尔佩特里埃医院);Beijing University of Technology(北京工业大学);Wuhan University(武汉大学);Wuhan Mental Health Center(武汉市精神卫生中心);Wuhan Hospital for Psychotherapy(武汉市心理医院)
Can Large Language Models be Used to Provide Psychological Counselling? An Analysis of GPT-4-Generated Responses Using Role-play Dialogues
The University of Electro-Communications(电气通信大学);iDEAR Human Support Service(iDEAR人文支持服务);Japanese Organization of Mental Health and Educational Agencies(日本心理健康与教育机构组织)
ChatCounselor: A Large Language Models for Mental Health Support
Hong Kong University(香港大学);Hong Kong University of Science and Technology(香港科技大学);The International Digital Economy Academy (IDEA)(国际数字经济研究院(IDEA));Ohio State University(俄亥俄州立大学)
EmoStage: A Framework for Accurate Empathetic Response Generation via Perspective-Taking and Phase Recognition
The University of Electro-Communications(电气通信大学);iDEAR Human Support Service(iDEAR人类支持服务);Japanese Organization of Mental Health and Educational Agencies(日本心理健康与教育机构组织)
Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human-LLM Collaboration
Ahsanullah University of Science and Technology(阿山努拉科技大学);Bangladesh University of Business and Technology(孟加拉国商业与技术大学);The University of New South Wales(新南威尔士大学);Jashore University of Science and Technology(贾绍尔科技大学);American International University - Bangladesh(孟加拉国美国国际大学)
DELTA: Deliberative Multi-Agent Reasoning with Reinforcement Learning for Multimodal Psychological Counseling
Anhui University(安徽大学);Hefei University of Technology(合肥工业大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合国家科学中心人工智能研究所)
Are Large Language Models Possible to Conduct Cognitive Behavioral Therapy?
School of Information Science and Engineering, Lanzhou University(兰州大学信息科学与工程学院);School of Medical Technology, Beijing Institute of Technology(北京理工大学医学技术学院)
DiaCBT: A Long-Periodic Dialogue Corpus Guided by Cognitive Conceptualization Diagram for CBT-based Psychological Counseling
Shanghai Institute of Artificial Intelligence for Education, East China Normal University(上海教育人工智能研究院,东华师范大学);School of Computer Science and Technology, East China Normal University(计算机科学与技术学院,东华师范大学);School of Psychology and Cognitive Science, East China Normal University(心理学与认知科学学院,东华师范大学)
PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation
School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)(合肥工业大学计算机与信息工程学院);Beijing Institute of Technology(北京理工大学);School of Medical Technology, Beijing Institute of Technology(北京理工大学医学技术学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Hefei Zhongjuyuan Intelligent Technology Co., Ltd.(合肥中聚源智能科技有限公司)
Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: A Benchmark Study
Indian Institute of Technology Delhi(印度理工学院德里分校);Indraprastha Institute of Information Technology Delhi(英德拉普拉斯塔信息技术学院);YourDOST;All India Institute of Medical Sciences, Rishikesh(全印医学科学学院里希盖什分校);All India Institute of Medical Sciences, New Delhi(全印医学科学学院新德里分校)
MIND: Towards Immersive Psychological Healing with Multi-agent Inner Dialogue
Sichuan University(四川大学);Shanghai Jiao Tong University(上海交通大学);Mental Health Center, West China Hospital, Sichuan University(四川大学西昌医院心理健康中心);WestChina School of Nursing, Sichuan University(四川大学西昌护理学院);National University of Singapore(新加坡国立大学)
Distinct Profiles of Run-to-Run Score Reliability and Expert-Panel Alignment Across Four LLM Evaluators of Simulated Japanese-Language AI-to-AI Counseling
Japan National Institute of Occupational Safety and Health(日本国立职业安全卫生研究所);Kaze To Taiyo(凯泽・太阳);Saga Occupational Health Association(Saga职业健康协会);Department of Pharmacy, Zikei Hospital/Zikei Institute of Psychiatry(药剂科,Zikei医院/Zikei精神医学研究所);Department of Medical Welfare, Suzuka University of Medical Science(医疗福祉科, Suzuka医科大学);Graduate School of Human Sciences, Ritsumeikan University(人类科学研究生院,立命馆大学);Faculty of Nursing, National Defense Medical College(护理学部,国家防卫医疗大学);Support Center for Students with Disabilities, Aoyama Gakuin University(残疾学生支持中心,上智大学)
CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs
Agency for Science, Technology and Research (A*STAR)(科技研究局);Nanyang Technological University(南洋理工大学);Hong Kong University of Science and Technology (Guangzhou), China(香港科技大学(广州),中国)
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
The Chinese University of Hong Kong(香港中文大学);Tianjin Medical University(天津医科大学);Tencent AI Lab(腾讯AI实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Jilin University(吉林大学);University of Science and Technology of China(中国科学技术大学);Shenzhen University of Advanced Technology(深圳理工大学);University of Macau(澳门大学);Huazhong University of Science and Technology(华中科技大学)
Structured Dialogue System for Mental Health: An LLM Chatbot Leveraging the PM+ Guidelines
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);East China Normal University(华东师范大学);Wuhan Research Institute of Posts and Telecommunications(武汉邮电科学研究院)
HEART-Bench: Do LLMs Make Persona-Consistent Decisions?
Shanghai Jiao Tong University(上海交通大学);Imperial College London(伦敦帝国理工学院);Quwan Group(启元集团);University of Washington(华盛顿大学);South China Normal University(华南师范大学)
Fostering Natural Conversation in Large Language Models with NICO: a Natural Interactive COnversation dataset
International Digital Economy Academy(国际数字经济学院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);Simon Fraser University(西蒙菲莎大学)
CNSocialDepress: A Chinese Social Media Dataset for Depression Risk Detection and Structured Analysis
Ertim Inalco;Milkuya Studio;Sorbonne Université(索邦大学);IRD Lab(IRD实验室);Faculty of Psychology, Peking University(北京大学心理学系);Faculty of Psychology and Cognitive Science, Beijing Normal University(北京师范大学心理学与认知科学系);Beijing Institute of Technology(北京理工大学)
The Expressions of Depression and Anxiety in Chinese Psycho-counseling: Usage of First-person Singular Pronoun and Negative Emotional Words
Hangzhou Normal University(杭州师范大学);Zhejiang University(浙江大学);Westlake University(西湖大学);Queen Mary University of London(伦敦玛丽女王大学);Hangzhou Polytechnic(杭州职业技术学院)