This is a pre-print of an article accepted to ACII 2019 conference (http://acii-conf.org/2019/). The final authenticated version will be published on IEEExplore as proceedings
The novel research area of computational empathy is in its infancy and moving towards developing methods and standards. One major problem is the lack of agreement on the evaluation of empathy in artificial interactive systems. Even though the existence of well-established methods from psychology, psychiatry and neuroscience, the translation between these methods and computational empathy is not straightforward. It requires a collective effort to develop metrics that are more suitable for interactive artificial agents. This paper is aimed as an attempt to initiate the dialogue on this important problem. We examine the evaluation methods for empathy in humans and provide suggestions for the development of better metrics to evaluate empathy in artificial agents. We acknowledge the difficulty of arriving at a single solution in a vast variety of interactive systems and propose a set of systematic approaches that can be used with a variety of applications and systems.
Can AI detect pain and express pain empathy? A review from emotion recognition and a human-centered AI perspective
Institute of Psychology, Chinese Academy of Sciences（中国科学院心理研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Oxford（牛津大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Hamburg（汉堡大学）；University of Macau（澳门大学）
Universal Empathy and Ethical Bias for Artificial General Intelligence
AIDEUS；National Research University of Information Technology, Mechanics and Optics（国立信息技术、机械与光学大学）；Aix Marseille Université（艾克斯-马赛大学）；CNRS（法国国家科学研究中心）
EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models
Fudan University（复旦大学）；School of Computer Science, Fudan University（复旦大学计算机科学学院）；Beijing Yuandian Technology（北京原点科技）；School of Engineering and Applied Science, University of Pennsylvania（宾夕法尼亚大学工程与应用科学学院）
FIGS: Evaluating Multi-Turn Sycophancy Without Penalizing Empathy
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；ELLIS Institute Tübingen（ELLIS图宾根研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（图宾根人工智能中心）
Your Robot Will Feel You Now: Empathy in Robots and Embodied Agents
School of Computing Science, Simon Fraser University（计算科学学院，西蒙弗雷泽大学）；School of Interactive Arts and Technology, Simon Fraser University（交互艺术与技术学院，西蒙弗雷泽大学）
From Visual Perception to Deep Empathy: An Automated Assessment Framework for House-Tree-Person Drawings Using Multimodal LLMs and Multi-Agent Collaboration
Tsinghua University（清华大学）；Shenzhen University（深圳大学）；University of Oxford（牛津大学）；Shenzhen Institute of Education Sciences（深圳市教育科学研究院）；Guangzhou University of Chinese Medicine（广州中医药大学）；Harbin Institute of Technology（哈尔滨工业大学）
Evaluating the Efficacy of Interactive Language Therapy Based on LLM for High-Functioning Autistic Adolescent Psychological Counseling
Branksome Hall Asia（亚洲布兰克森霍尔学院）；Algorix Convergence Research Office（Algorix融合研究室）；Korea Science Academy of KAIST（韩国科学技术院附属韩国科学英才学校）；Hana Academy Seoul（首尔汉拿学院）
Polycraft World AI Lab (PAL): An Extensible Platform for Evaluating Artificial Intelligence Agents
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Georgia Institute of Technology and Emory University（佐治亚理工学院和埃默里大学）
随着人工智能研究的进步，用于评估人工智能智能体的平台需要适应和发展，以继续挑战它们。我们介绍了Polycraft World AI Lab（PAL），这是一个基于Minecraft mod Polycraft World的API任务模拟器。我们的平台旨在允许具有不同架构的人工智能智能体轻松地与Minecraft世界进行交互，训练并在多个任务中进行评估。PAL允许以灵活的方式创建任务，并能够在评估期间操纵任务的任何方面。人工智能智能体和外部参与者（非玩家角色，NPC）在开放世界环境中采取的所有行动都被记录下来，以简化评估。在这里，我们提出了两个自定义任务的PAL平台上，一个专注于多步规划和一个专注于导航，并评估代理解决这些问题。总之，我们报告了一个多功能和可扩展的人工智能评估平台，人工智能研究人员可以利用它的低门槛。
Dungeon Crawl Stone Soup as an Evaluation Domain for Artificial Intelligence
Naval Research Laboratory（海军研究实验室）；Knexus Research Corporation（奈克瑟斯研究公司）
Dungeon Crawl Stone Soup是一款流行的单人游戏，免费和开源的Rogue类视频游戏，具有足够复杂的决策空间，使其成为认知系统研究的理想测试平台，更广泛地说，人工智能。本文介绍了地下城爬行石头汤的属性，有利于评估人工智能系统的新方法。我们还强调了为AI研究人员构建API的持续努力，这些API体现了最近的游戏API，如MALMO，ELF和Starcraft II API。Dungeon Crawl Stone Soup的复杂性为评估AI和认知系统提供了重要的机会，包括人类用户研究。在本文中，我们提供了（1）地下城爬行石头汤的状态空间的描述，（2）我们的API组件的描述，以及（3）在地下城爬行石头汤视频游戏中评估人工智能智能体的潜在好处。
Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training
The Ohio State University（俄亥俄州立大学）；The Hong Kong University of Science and Technology（香港科技大学）；Southern University of Science and Technology（南方科技大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Guangzhou Medical University（广州医科大学）
Explainable Artificial Intelligence Model for Evaluating Shear Strength Parameters of Municipal Solid Waste Across Diverse Compositional Profiles
The Joint Graduate School of Energy and Environment, King Mongkut's University of Technology Thonburi（泰国国王科技大学吞武里校区能源与环境联合研究生院）；AI Research Group, Department of Civil Engineering, Faculty of Engineering, King Mongkut's University of Technology Thonburi（泰国国王科技大学吞武里校区工程学院土木工程系人工智能研究组）；Center of Excellence on Energy Technology and Environment (CEE), PERDO, Ministry of Higher Education, Science, Research and Innovation (MHESI)（高等教育、科学、研究与创新部PERDO能源技术与环境卓越中心）；Department of Environmental Engineering, Faculty of Engineering, Kasetsart University（农业大学工程学院环境工程系）
Evaluating Single and Multi-Omics Based Explainable Artificial Intelligence (MOXAI) for Molecular Subclass Classification of Adult-Type Diffuse Gliomas
Evaluating the Effectiveness of Artificial Intelligence in Predicting Adverse Drug Reactions among Cancer Patients: A Systematic Review and Meta-Analysis
National Polytechnic School of Oran - Maurice Audin（奥兰国家理工学院 - 莫里斯·奥丹）
The Animal-AI Environment: Training and Testing Animal-Like Artificial Cognition
Imperial College London（帝国理工学院）；Leverhulme Centre for the Future of Intelligence（利弗休姆未来智能中心）；Universitat Politècnica de València（瓦伦西亚理工大学）；University of Cambridge（剑桥大学）
Prepared Or Unprepared? Evaluating Healthcare Workforce Readiness for Clinical Adoption of Artificial Intelligence in Nigeria
College of Health Sciences, Bayero University Kano（贝耶罗大学卡诺分校健康科学学院）；Federal University of Health Sciences, Azare（阿扎雷联邦健康科学大学）；African Institute for Research Advancement & Innovation（非洲研究促进与创新研究所）；Medical Artificial Intelligence Laboratory (MAI Lab)（医学人工智能实验室）；Aminu Kano Teaching Hospital（阿米努·卡诺教学医院）；Nnamdi Azikiwe University Teaching Hospital（纳姆迪·阿齐基韦大学教学医院）；Lagos State Teaching Hospital（拉各斯州立教学医院）
A multidisciplinary task-based perspective for evaluating the impact of AI autonomy and generality on the future of work
Joint Research Centre, European Commission（欧盟委员会联合研究中心）；Universitat Pompeu Fabra（庞培法布拉大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；University of Cambridge（剑桥大学）
An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
NUS Department of Computer Science（国立新加坡大学计算机科学系）；MIT EECS（麻省理工学院电子工程与计算机科学系）；A*STAR（新加坡科技研究局）；Singapore-MIT Alliance for Research and Technology (SMART)（新加坡-麻省理工联合研究技术机构（SMART））
