To advance the evaluation of multimodal math reasoning in large multimodal models (LMMs), this paper introduces a novel benchmark, MM-MATH. MM-MATH consists of 5,929 open-ended middle school math problems with visual contexts, with fine-grained classification across difficulty, grade level, and knowledge points. Unlike existing benchmarks relying on binary answer comparison, MM-MATH incorporates both outcome and process evaluations. Process evaluation employs LMM-as-a-judge to automatically analyze solution steps, identifying and categorizing errors into specific error types. Extensive evaluation of ten models on MM-MATH reveals significant challenges for existing LMMs, highlighting their limited utilization of visual information and struggles with higher-difficulty problems. The best-performing model achieves only 31% accuracy on MM-MATH, compared to 82% for humans. This highlights the challenging nature of our benchmark for existing models and the significant gap between the multimodal reasoning capabilities of current models and humans. Our process evaluation reveals that diagram misinterpretation is the most common error, accounting for more than half of the total error cases, underscoring the need for improved image comprehension in multimodal reasoning.
Fine-Grained Analysis of Propaganda in News Articles
Qatar Computing Research Institute, HBKU(哈马德·本·哈利法大学卡塔尔计算研究所);MIT Computer Science and Artificial Intelligence Laboratory(麻省理工学院计算机科学与人工智能实验室);Università di Bologna(博洛尼亚大学);A Data Pro
EmoNet-Voice: A Fine-Grained, Expert-Verified Benchmark for Speech Emotion Detection
LAION e.V.(LAION研究院);TU Munich(慕尼黑技术大学);L3S Research Center(L3S研究中心);Leibniz University of Hannover(汉诺威莱布尼茨大学);TU Darmstadt(达姆施塔特技术大学);Ontocord(Ontocord公司);DFKI(DFKI研究所);TIB–Leibniz Information Centre for Science & Technology(TIB-莱布尼茨科学与技术信息中心)
Advancing Visual Grounding with Scene Knowledge: Benchmark and Method
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Sun Yat-sen University(中山大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);Tencent AI Lab(腾讯人工智能实验室);AI 3 Institute, Fudan University(复旦大学人工智能创新与产业研究院)
Language Representation Models for Fine-Grained Sentiment Classification
Cornell Tech(康奈尔科技学院)
情感分类是一个快速发展的研究领域,几乎在任何领域都有应用。虽然各种模型和数据集在二进制分类任务中表现出很高的准确性,但细粒度情感分类任务仍然是一个有显着改进空间的领域。分析SST-5数据集,Munikar等人(2019)的先前工作表明,嵌入工具BERT允许一个简单的模型实现最先进的准确性。自那篇论文发表以来,已经发表了几种BERT替代方案,其中三种主要方案是Al BERT(Lan等人,2019)、DistilBERT(Sanh et al. 2019)和RoBERTa(Liu et al. 2019年)。虽然这些模型在流行的基准测试GLUE、SQuAD和RACE上报告了比BERT的一些改进,但它们尚未应用于细粒度分类任务。在本文中,我们通过复制Munikar等人的BERT模型,以及将嵌入层交换到备选模型。在实验中,我们发现AlBERT遭受的准确性损失比其他任务报告的要大得多,DistilBERT的准确性损失与其他任务报告的损失相似,同时是训练最快的模型,RoBERTa在SST-5根水平上达到了最新的预测准确性(60.2%)。
FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning
Department of Computer Science, University of Texas at Dallas(德克萨斯大学达拉斯分校计算机科学系);Department of Electrical Engineering Computer Science, University of Michigen(密歇根大学电子工程与计算机科学系)
Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentation
College of Intelligence and Computing, Tianjin University(智能与计算学院,天津大学);NJUST(南京理工大学);College of Mathematics, Taiyuan University of Technology(数学学院,太原科技大学);Tianjin University of Science and Technology(天津科技大学)
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages
Sharif University of Technology(谢里夫理工学院);Vellore Institute of Technology(韦洛雷理工学院);IIT Kharagpur(印度理工学院达卡分校);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Advancing AI Research Assistants with Expert-Involved Learning
Yale University(耶鲁大学);Broad Institute of MIT and Harvard(麻省理工学院-哈佛大学博德研究所);Google DeepMind(谷歌DeepMind);Stanford University(斯坦福大学);Genentech(基因泰克);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Cornell University(康奈尔大学);Harvard University(哈佛大学)
Advancing Multi-Step Mathematical Reasoning in Large Language Models through Multi-Layered Self-Reflection with Auto-Prompting
Luiz de Queiroz College of Agriculture University of São Paulo(圣保罗大学农业学院);School of Engineering and Sciences Tecnologico de Monterrey(蒙特雷技术学院工程与科学学院);Institute of Mathematics and Computer Sciences University of São Paulo(圣保罗大学数学与计算机科学研究所)
$A^2R^2$: Advancing Img2LaTeX Conversion via Visual Reasoning with Attention-Guided Refinement
University of California, San Diego(加州大学圣地亚哥分校);ByteDance(字节跳动);University of California, Merced(加州大学默塞德分校);University of Southern California(南加州大学);University at Buffalo(布法罗大学);The University of Queensland(昆士兰大学)
PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Soochow University(苏州大学);Stony Brook University(石溪大学);The Chinese University of Hong Kong(香港中文大学)
Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning
Fudan University(复旦大学);XingYun lab, HUJING Digital Media & Entertainment Group(星云实验室,HUJING数字媒体与娱乐集团);University of Science and Technology Beijing(北京科技大学);Chinese Academy of Sciences(中国科学院);Beijing University of Posts and Telecommunications(北京邮电大学)
A self-evolving multi-role collaborative framework with fine-grained difficulty guidance for innovative mathematical problem generation
School of Physics and Information Technology, Shaanxi Normal University(陕西师范大学物理与信息技术学院);Department of Computing Technologies, Swinburne University of Technology(斯威本科技大学计算技术系);Department of Software Engineering, University of Szeged(塞格德大学软件工程系)