Evaluation of large language models (LLMs) has raised great concerns in the community due to the issue of data contamination. Existing work designed evaluation protocols using well-defined algorithms for specific tasks, which cannot be easily extended to diverse scenarios. Moreover, current evaluation benchmarks can only provide the overall benchmark results and cannot support a fine-grained and multifaceted analysis of LLMs' abilities. In this paper, we propose meta probing agents (MPA), a general dynamic evaluation protocol inspired by psychometrics to evaluate LLMs. MPA is the key component of DyVal 2, which naturally extends the previous DyVal~\citep{zhu2023dyval}. MPA designs the probing and judging agents to automatically transform an original evaluation problem into a new one following psychometric theory on three basic cognitive abilities: language understanding, problem solving, and domain knowledge. These basic abilities are also dynamically configurable, allowing multifaceted analysis. We conducted extensive evaluations using MPA and found that most LLMs achieve poorer performance, indicating room for improvement. Our multifaceted analysis demonstrated the strong correlation between the basic abilities and an implicit Matthew effect on model size, i.e., larger models possess stronger correlations of the abilities. MPA can also be used as a data augmentation approach to enhance LLMs. Code is available at: https://github.com/microsoft/promptbench.
Georgia Institute of Technology（佐治亚理工学院）；DeepMind（深度思维）
Recent work has shown how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments. We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling -action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019). After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we p…
School of Electrical and Computer Engineering（电气与计算机工程学院）；College of Engineering（工程学院）；University of Tehran（德黑兰大学）
Human languages are full of metaphorical expressions. Metaphors help people understand the world by connecting new concepts and domains to more familiar ones. Large pre-trained language models (PLMs) are therefore assumed to encode metaphorical knowledge useful for NLP systems. In this paper, we investigate this hypothesis for PLMs, by probing metaphoricity information in their encodings, and by measuring the cross-lingual and cross-dataset generalization of this information. We present studies in multiple metaphor detection datasets and in four languages (i.e., English, Spanish, Russian, and …
SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Zhongguancun Academy（中关村学院）；Tsinghua University（清华大学）；Helsinki University（赫尔辛基大学）
SocialVeil: Probing Social Intelligence of Language Agents under Communication Barriers
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校Siebel计算与数据科学学院）；Department of Psychological Sciences, Rice University（里士满大学心理学系）
Probe Before You Edit: Probing-Guided Molecular Optimization for LLM Agents in Structure-Based Drug Design
The Hong Kong University of Science and Technology（香港科学与技术大学）；City University of Hong Kong（香港城市大学）；Beijing Institute of Mathematical Sciences and Applications（北京数学科学应用研究所）
Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase
Faculty of Psychology, University of Basel（巴塞尔大学心理学系）；Center for Adaptive Rationality, Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所适应性理性中心）；School of Psychology, UNSW（新南威尔士大学心理学学院）
Beyond Completion: Probing Cumulative State Tracking to Predict LLM Agent Performance
Tohoku University（东北大学）；Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学）；Texas A&M University（德克萨斯农工大学）；Worcester Polytechnic Institute（伍斯特理工学院）
CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Games
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education（教育部智能技术与教育应用工程研究中心）
MetaEvo: A Meta-Optimization Framework for Experience-Driven Agent Evolution
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Beijing Institute of Technology Southeast Academy of Information Technology（北京理工大学东南信息技术研究院）
The Epi-LLM Framework: probing LLM behavioral priors through epidemiological agent-based models
Big Data Institute, Li Ka Shing Center for Health Information and Discovery, University of Oxford, Oxford, United Kingdom（大数据研究所、李嘉诚健康信息与发现中心、牛津大学、牛津、英国）；Leverhulme Centre for Demographic Science, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom（勒弗赫姆人口科学中心、努尔菲尔德人口健康系、牛津大学、牛津、英国）；Pandemic Sciences Institute, Nuffield Department of Medicine, University of Oxford, Oxford, United Kingdom（流行病学科学研究所、努尔菲尔德医学系、牛津大学、牛津、英国）；Department of Genomics and Computational Biology, UMass Chan Medical School, United States（基因组与计算生物学系、UMass Chan医学学校、美国）；Broad Institute of Harvard and MIT, United States（哈佛大学和麻省理工学院Broad研究所、美国）；The Queen’s College, University of Oxford, Oxford, United Kingdom（女王学院、牛津大学、牛津、英国）
The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Ant Group（蚂蚁集团）
