It is increasingly common to evaluate the same coreference resolution (CR) model on multiple datasets. Do these multi-dataset evaluations allow us to draw meaningful conclusions about model generalization? Or, do they rather reflect the idiosyncrasies of a particular experimental setup (e.g., the specific datasets used)? To study this, we view evaluation through the lens of measurement modeling, a framework commonly used in the social sciences for analyzing the validity of measurements. By taking this perspective, we show how multi-dataset evaluations risk conflating different factors concerning what, precisely, is being measured. This in turn makes it difficult to draw more generalizable conclusions from these evaluations. For instance, we show that across seven datasets, measurements intended to reflect CR model generalization are often correlated with differences in both how coreference is defined and how it is operationalized; this limits our ability to draw conclusions regarding the ability of CR models to generalize across any singular dimension. We believe the measurement modeling framework provides the needed vocabulary for discussing challenges surrounding what is actually being measured by CR evaluations.
Anatomy of OntoGUM--Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms
Georgetown University（乔治城大学）；Linguistic Data Consortium（语言数据联盟）；University of Pennsylvania（宾夕法尼亚大学）；Department of Linguistics, Georgetown University（乔治城大学语言学系）
CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
The Hong Kong Polytechnic University（香港理工大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Research Centre for Data Science & Artificial Intelligence（数据科学与人工智能研究中心）
Evaluating and Improving the Coreference Capabilities of Machine Translation Models
The Hebrew University of Jerusalem（希伯来大学）；Bar Ilan University（巴伊兰大学）
机器翻译（MT）需要广泛的语言能力，当前的端到端模型期望通过观察双语语料库中对齐的句子来隐式地学习这些语言能力。在这项工作中，我们要求：\emph{How well do MT models learn coreference resolution from implicit signal?}为了回答这个问题，我们开发了一种评估方法，从MT输出中获得共指聚类，并在不需要目标语言注释的情况下对其进行评估。我们进一步评估了几个著名的开源和商业MT系统，从英语翻译到六种目标语言，并将它们与三个具有挑战性的基准测试中最先进的共指消解器进行比较。我们的研究结果表明，单语解析器大大优于MT模型。受此结果的启发，我们实验了不同的方法，将共指消解模型的输出纳入MT中，显示出对强基线的改进。
Evaluating Discourse Phenomena in Neural Machine Translation
University of Zurich（苏黎世大学）；LIMSI, CNRS, Univ. Paris-Sud, Université Paris-Saclay（法国国家科学研究中心、巴黎第十一大学、巴黎萨克雷大学联合计算机科学与数学应用实验室）；University of Edinburgh（爱丁堡大学）
Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective
Harbin Institute of Technology（哈尔滨工业大学）；Central South University（中南大学）；Chinese Academy of Sciences（中国科学院）；Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；University of Toronto（多伦多大学）；Zhili College, Tsinghua University（清华大学至理书院）
Evaluating Large Language Model Creativity from a Literary Perspective
Imperial College London（帝国理工学院）；Google DeepMind（谷歌DeepMind）；Institute of Historical Research, School of Advanced Study, University of London（伦敦大学高级研究院历史研究所）；University of Southampton（南安普顿大学）
Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全国家重点实验室，计算技术研究所）；Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院网络数据科学与技术重点实验室，计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics
School of Architecture, Tsinghua University（清华大学建筑学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；BNRist, Tsinghua University（清华大学BNRist）
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective
Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）；Central South University（中南大学）；University of Electronic Science and Technology of China（电子科技大学）；Microsoft Research Asia（微软亚洲研究院）
SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions
The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）；Johns Hopkins University（约翰霍普金斯大学）；University of California Irvine（加州大学尔湾分校）；Stanford University（斯坦福大学）
A Multi-Perspective Benchmark and Moderation Model for Evaluating Safety and Adversarial Robustness
Department of Computer and Information Science, Fordham University（福德汉大学计算机与信息科学系）；College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·卡西姆大学科学与工程学院）；Department of Computer Science and Engineering, Qatar University（卡塔尔大学计算机科学与工程系）
WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Models
Information Technology University Department of Computer Science（信息科技大学计算机科学系）；Zayed University College of Interdisciplinary Studies（扎耶德大学跨学科研究学院）；Computer Science and Engineering Department（计算机科学与工程系）
