Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground
Department of Computer Science(计算机科学系);Department of Linguistics(语言学系);Institute for Advanced Computational Science(高级计算科学研究院);Stony Brook University(石溪大学);University of Guilan(吉兰大学)
Adil Soubki, John Murzaku, Arash Yousefi Jordehi, Peter Zeng, Magdalena Markowska, Seyed Abolghasem Mirroshandel, Owen Rambow
Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received a great deal of attention. However, many existing benchmarks rely on synthetic data, which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on Common-ToM.
MiraMind: Benchmarking Reliable Mental Health Reasoning beyond Answer Accuracy
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);Center for Language and Information Research, Wuhan University(武汉大学语言信息研究中心);The University of Manchester(曼彻斯特大学);School of Computer Science, Wuhan University(武汉大学计算机科学学院);Mount Holyoke College(马歇尔学院);Hefei University of Technology(合肥工业大学)
Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands(医学信息学系,阿姆斯特丹大学医学中心,阿姆斯特丹大学,阿姆斯特丹,荷兰);Amsterdam Public Health, Methodology, Amsterdam, The Netherlands(阿姆斯特丹公共健康,方法学,阿姆斯特丹,荷兰)
Inducing language models to assert their own consciousness restores human beliefs and values
Google(谷歌);University of Chicago(芝加哥大学);University of London(伦敦大学);University of Washington(华盛顿大学);Northwestern University(西北大学);Santa Fe Institute(圣达菲研究所)
ToMBench: Benchmarking Theory of Mind in Large Language Models
Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);IIE, CAS(中国科学院自动化研究所);Tianjin University(天津大学);Northwest Minzu University(西北民族大学);Nankai University(南开大学);Beijing Institute of Technology(北京理工大学)
Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning
Hebrew University of Jerusalem(海法大学);MPI for Biological Cybernetics(生物感知研究所);Centre for AI and Machine Learning, AU(人工智能与机器学习中心);IoPPN, King’s College London(国王学院伦敦人工智能与感知神经科学中心)
AnthroDial: Benchmarking LLM Anthropomorphism in Autonomous Social Interaction
Shanghai Institute of Innovation(上海创新研究院);University of Science and Technology of China(中国科学技术大学);East China Normal University(华东师范大学);Anhui University(安徽大学);Shanghai Jiaotong University(上海交通大学);Fudan University(复旦大学);University of Melbourne(墨尔本大学);Zhejiang University(浙江大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shanghai Tianyou Software Co., Ltd.(上海天游软件有限公司);Chabiyue (Shanghai) Information Technology Co., Ltd.(茶百悦(上海)信息技术有限公司);Zhejiang Century Huatong Group Co., Ltd.(浙江世纪华通集团股份有限公司)
Theory of Mind and Self-Attributions of Mentality are Dissociable in LLMs
Google, Paradigms of Intelligence Team(Google,智能范式团队);Knowledge Lab, University of Chicago(芝加哥大学知识实验室);Institute of Philosophy, School of Advanced Study, University of London(伦敦大学高等研究院哲学研究所);Department of Biomedical Informatics and Medical Education and Department of Bioethics and Humanities, School of Medicine, University of Washington(华盛顿大学医学院生物医学信息学与医学教育系及生物伦理学与人文学系);Kellogg School of Management, Northwestern University(西北大学凯洛格管理学院);Santa Fe Institute(圣塔菲研究所)
Mind Your POV: Convergence of Articles and Editors Towards Wikipedia's Neutrality Norm
Georgia Institute of Technology(佐治亚理工学院)
维基百科有一个很强的“中立观点”(Neutral Point of View,缩写为NPR)的写作规范。违反这一规范的文章会被标记,并鼓励编辑进行更正。但是,这种标记系统的影响还没有得到定量测量。Numbers标签是否有助于文章收敛到所需的风格?Numbers的更正是否鼓励编辑采用这种风格?我们研究这些问题使用语料库的NPOV标记的文章和一组词汇与偏见的语言。一个中断的时间序列分析表明,在一篇文章被标记为Numbers后,文章中有偏见的语言显著减少,这是由几个词汇测量的。然而,对于个人编辑来说,Numbers的更正和讨论页面的讨论并没有对大多数这些词汇中的单词的用法产生显著的变化,包括维基百科自己的“要注意的单词”列表。这表明,Numbers标签和讨论确实改善了内容,但在让编辑适应网站的语言规范方面却不太成功。
I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Southern California(南加州大学);University of Washington(华盛顿大学);University of Pennsylvania(宾夕法尼亚大学);MIT(麻省理工学院)