In this paper I describe and reduce to practice an objective protocol for evaluating the cognitive capabilities of a non-human system against human cognition in a laboratory environment. This is important because the existence of a non-human system with cognitive capabilities comparable to those of humans might make once-philosophical questions of safety and ethics immediate and urgent. Past attempts to devise evaluation methods, such as the Turing Test and many others, have not met this need; most of them either emphasize a single aspect of human cognition or a single theory of intelligence, fail to capture the human capacity for generality and novelty, or require success in the physical world. The protocol is broadly Bayesian, in that its primary output is a confidence statistic in relation to a claim. Further, it provides insight into the areas where and to what extent a particular system falls short of human cognition, which can help to drive further progress or precautions.
The Animal-AI Environment: A Virtual Laboratory For Comparative Cognition and Artificial Intelligence Research
University of Cambridge（剑桥大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；University of Hull（赫尔大学）；University of Exeter（埃克塞特大学）；Imperial College London（伦敦帝国学院）
Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships
Princeton University（普林斯顿大学）；University of Cambridge（剑桥大学）；Leverhulme Centre for Future Intelligence（勒沃胡姆未来智能中心）；Universitat Politècnica de València（瓦伦西亚理工大学）；The Alan Turing Institute（艾伦·图灵研究所）；NYU（纽约大学）
Not Yet AlphaFold for the Mind: Evaluating Centaur as a Synthetic Participant
Institute of Cognitive Science（认知科学研究所）；Osnabrück University（奥斯纳布鲁克大学）；Princeton University（普林斯顿大学）；Brown University（布朗大学）；Robert J. and Nancy D. Carney Institute for Brain Science（罗伯特·J．和南西·D．卡尼脑科学研究所）；Department of Neuroscience（神经科学系）；Department of Cognitive and Psychological Sciences（认知与心理学系）
A word association network methodology for evaluating implicit biases in LLMs compared to humans
University of Pisa, Department of Computer Science（比萨大学计算机科学系）；National Research Council of Italy, Institute of Information Science and Technologies（意大利国家研究委员会信息科学与技术研究所）；University of Trento, Department of Psychology and Cognitive Science（特伦托大学心理学与认知科学系）
CityGPT: Empowering Urban Spatial Cognition of Large Language Models
Department of Electronic Engineering, BNRist, Tsinghua University（电子工程系、BNRist、清华大学）；School of Electronic and Information Engineering, Beijing Jiaotong University（电子信息工程学院、北京交通大学）；Department of Electronic Engineering, Tsinghua University（电子工程系、清华大学）；Department of Urban Planning, Tsinghua University（城市规划系、清华大学）
M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmark
AutoLab, Westlake University（西湖大学AutoLab）；Alibaba Group（阿里巴巴集团）；Zhejiang University（浙江大学）；Key Laboratory of Behavioral Science, Institute of Psychology, CAS（中国科学院心理研究所行为科学重点实验室）；Key Laboratory of AI Safety, Institute of Computing Technology, CAS（中国科学院计算技术研究所人工智能安全重点实验室）
AwarenessBench: Assessing Cognitive Capabilities of Language Models
Tsinghua University（清华大学）；Shanghai Qi Zhi Institute（上海期智研究院）；Carnegie Mellon University（卡内基梅隆大学）；Columbia University（哥伦比亚大学）；Fangcun AI（方寸智能）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）
Evaluating alignment between humans and neural network representations in image-based learning tasks
Institute for Human-Centered AI, Helmholtz Computational Health Center（亥姆霍兹计算健康中心以人为本人工智能研究所）；Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）；Max Planck School of Cognition（马克斯·普朗克认知科学学院）；Max Planck Institute for Human Cognitive & Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；University of Tübingen（蒂宾根大学）；Kavli Institute for Systems Neuroscience（卡夫利系统神经科学研究所）；Leipzig University（莱比锡大学）；Technical University Dresden（德累斯顿工业大学）；Julius-Maximilians-Universität Würzburg（维尔茨堡大学）；Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所）
Cross-subject Brain Functional Connectivity Analysis for Multi-task Cognitive State Evaluation
School of Electronics and Information, Northwestern Polytechnical University（西北工业大学电子信息学院）；Chongqing Institute for Brain and Intelligence（重庆脑与智能研究院）；Guangyang Bay Laboratory（广阳湾实验室）；King Abdullah II School of Information Technology, The University of Jordan（约旦大学阿卜杜拉二世信息技术学院）；School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；School of Computing, SRM University（SRM大学计算机学院）；School of Engineering, The Amity University（阿米提大学工程学院）；School of Aeronautics, Northwestern Polytechnical University（西北工业大学航空学院）
RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Faculty of Psychology and Educational Sciences, Alexandru Ioan Cuza University of Iași（亚历山德鲁伊万库扎大学心理学与教育科学学院）；School of Electronic Information, Wuhan University（武汉大学电子信息学院）；School of Public Health, The University of Hong Kong（香港大学公共卫生学院）；School of Computer Science, The University of Auckland（奥克兰大学计算机科学学院）；Department of Data Science and Artificial Intelligence, Monash University（莫纳什大学数据科学与人工智能系）
