Commonsense question-answering (QA) tasks, in the form of benchmarks, are constantly being introduced for challenging and comparing commonsense QA systems. The benchmarks provide question sets that systems' developers can use to train and test new models before submitting their implementations to official leaderboards. Although these tasks are created to evaluate systems in identified dimensions (e.g. topic, reasoning type), this metadata is limited and largely presented in an unstructured format or completely not present. Because machine common sense is a fast-paced field, the problem of fully assessing current benchmarks and systems with regards to these evaluation dimensions is aggravated. We argue that the lack of a common vocabulary for aligning these approaches' metadata limits researchers in their efforts to understand systems' deficiencies and in making effective choices for future tasks. In this paper, we first discuss this MCS ecosystem in terms of its elements and their metadata. Then, we present how we are supporting the assessment of approaches by initially focusing on commonsense benchmarks. We describe our initial MCS Benchmark Ontology, an extensible common vocabulary that formalizes benchmark metadata, and showcase how it is supporting the development of a Benchmark tool that enables benchmark exploration and analysis.
Com$^2$: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
Research Center for Social Computing and Interactive Robotics（社会计算与交互机器人研究中心）；Harbin Institute of Technology（哈尔滨工业大学）；Institute of Trustworthy Embodied AI（可信具身人工智能研究院）；Fudan University（复旦大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Nanjing University（南京大学）
DORA: Exploring Outlier Representations in Deep Neural Networks
Leibniz Institute for Agriculture and Bioeconomy (ATB)（莱布尼茨农业与生物经济研究所（ATB））；Technical University of Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））；Korea University（高丽大学）；Max Planck Institut für Informatik（马克斯·普朗克信息学研究所）；Google Research（谷歌研究院）；University of Potsdam（波茨坦大学）；UiT Arctic University of Norway（挪威北极大学）
Brain Structure-Function Fusing Representation Learning using Adversarial Decomposed-VAE for Analyzing MCI
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen University（深圳大学）；Maebashi Institute of Technology（前桥工业大学）
