A `state of the art' model A surpasses humans in a benchmark B, but fails on similar benchmarks C, D, and E. What does B have that the other benchmarks do not? Recent research provides the answer: spurious bias. However, developing A to solve benchmarks B through E does not guarantee that it will solve future benchmarks. To progress towards a model that `truly learns' an underlying task, we need to quantify the differences between successive benchmarks, as opposed to existing binary and black-box approaches. We propose a novel approach to solve this underexplored task of quantifying benchmark quality by debuting a data quality metric: DQI.
EEG Benchmarking Needs a Task Specification Layer: NeuroDoc for Rulebook-Guided, Executable Benchmark Construction
School of Advanced Technology, Xi’an Jiaotong-Liverpool University(西交利物浦大学先进技术学院);School of Electrical Engineering, Electronics and Computer Science, University of Liverpool(利物浦大学电气工程、电子与计算机科学学院);School of Computer Science and Informatics, University of Liverpool(利物浦大学计算机科学与信息学学院);Department of Data Science and Artificial Intelligence, Hong Kong Polytechnic University(香港理工大学数据科学与人工智能系)
Costal Cartilage Segmentation with Topology Guided Deformable Mamba: Method and Benchmark
Plastic Surgery Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College(中国医学科学院北京协和医学院整形外科医院);School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)理工学院);Nanjing University of Science and Technology(南京理工大学);The Second Hospital of Hebei Medical University(河北医科大学第二医院)
Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching
School of Computing, National University of Singapore(新加坡国立大学计算机学院);FST and ICI, University of Macau(澳门大学科技学院与当代计算机与软件研究所);School of Communication Engineering, Hangzhou Dianzi University(杭州电子科技大学通信工程学院)
Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
EduGage: A Multimodal Dataset and Benchmark for Sensor-Based Momentary Assessment of Engagement in Self-Guided Video Learning
School of Interactive Computing, Georgia Institute of Technology(佐治亚理工学院交互计算学院);College of Computing, Georgia Institute of Technology(佐治亚理工学院计算机学院);School of Computing Technologies, RMIT University(皇家墨尔本理工大学计算技术学院)
GuideWeb: A Benchmark for Automatic In-App Guide Generation on Real-World Web UIs
Techtouch Inc.(Techtouch公司);University of Chinese Academy of Sciences(中国科学院大学);Shenzhen Institute of Advanced Technology Chinese Academy of Sciences(中国科学院深圳先进技术研究所);Yokohama National University(Yokohama国立大学)
MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
University of Chinese Academy of Sciences(中国科学院大学);Monash University(莫纳什大学);Pusan National University(釜山国立大学);Shenzhen University of Advanced Technology(深圳先进技术大学)
Event Guided Denoising for Multilingual Relation Learning
Columbia University(哥伦比亚大学)
通用关系抽取最近取得了相当大的进展,部分原因是Soares et al.(2019)的大规模数据密集型远程监督技术,该技术在许多基准测试中产生了最先进的结果。在这项工作中,我们提出了一种方法,用于收集高质量的训练数据,从未标记的文本中关系抽取,实现了近重建的zero-shot和Few-Shot的结果在一小部分的训练成本。我们的方法利用可预测的分布结构的日期标记的新闻文章,建立一个去噪语料库-提取过程过滤掉低质量的例子。我们表明,在这个语料库上训练的一个较小的多语言编码器在英语和西班牙语的Few-Shot和标准关系基准上执行了与当前最先进水平(当两者都很少或没有微调时)的匹配,尽管使用了更少的示例(50 k与300 mil+)。
Com$^2$: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究中心);Harbin Institute of Technology(哈尔滨工业大学);Institute of Trustworthy Embodied AI(可信具身人工智能研究院);Fudan University(复旦大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Nanjing University(南京大学)