MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
National University of Singapore（新加坡国立大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；Hong Kong Baptist University（香港浸会大学）
Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available at https://github.com/likaixin2000/MMCode.
Benchmarking Interaction, Beyond Policy: a Reproducible Benchmark for Collaborative Instance Object Navigation
Sapienza University of Rome（罗马大学）；University of Verona（维罗纳大学）；Polytechnic of Turin（都灵理工大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；Reykjavik University（雷克雅未克大学）
ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models
School of Data Science, Fudan University（复旦大学数据科学学院）；Zhengzhou Zhongke Institute of Integrated Circuit and System Application（郑州中凯集成电路与系统应用研究院）；School of Computer Science, Fudan University（复旦大学计算机科学学院）
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China（计算机学院，国家多媒体软件工程技术研究中心和湖北多媒体与网络通信工程重点实验室，武汉大学，中国）；Alibaba Group, Hangzhou, China（阿里巴巴集团，杭州，中国）；Independent Researcher（独立研究者）；Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates（机器学习系，Mohamed bin Zayed人工智能大学，阿拉伯联合酋长国）
SeqBench: Benchmarking Sequential Narrative Generation in Text-to-Video Models
University of Michigan（密歇根大学）；Northeastern University（东北大学）；University of Washington（华盛顿大学）；Harvard University（哈佛大学）；Beijing Jiaotong University（北京交通大学）；University of Rochester（罗切斯特大学）；School of Earth Sciences（地球科学学院）
PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；ModelBest Inc.（ModelBest公司）；State Key Lab. of Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Institute for AI, Tsinghua University（清华大学人工智能研究院）
Benchmarking Vision-Language and Multimodal Large Language Models in Zero-shot and Few-shot Scenarios: A study on Christian Iconography
Gianmarco Spinaci Department of Classical Philology and Italian Studies, University of Bologna, Italy Villa i Tatti, The Harvard University Center for Italian Renaissance Studies, Florence, Italy（Gianmarco Spinaci 文艺复兴研究系，博洛尼亚大学，意大利 塔蒂别墅，哈佛大学意大利文艺复兴研究中心，佛罗伦萨，意大利）；Lukas Klic Villa i Tatti, The Harvard University Center for Italian Renaissance Studies, Florence, Italy（Lukas Klic 塔蒂别墅，哈佛大学意大利文艺复兴研究中心，佛罗伦萨，意大利）；Giovanni Colavizza Department of Classical Philology and Italian Studies, University of Bologna, Italy Department of Communication, University of Copenhagen, Denmark（Giovanni Colavizza 文艺复兴研究系，博洛尼亚大学，意大利 传播系，哥本哈根大学，丹麦）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；ICSI（国际计算机科学研究所）；UC Berkeley（加州大学伯克利分校）；Disney Research（迪士尼研究中心）；École Polytechnique de Montréal（蒙特利尔理工学院）；Université de Sherbrooke（舍布鲁克大学）；Twitter（推特）；Université de Montréal（蒙特利尔大学）
Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）；Sun Yat-Sen University（中山大学）；Chinese University of Hong Kong（香港中文大学）
Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extraction
School of Computer Science, Fudan University（复旦大学计算机科学学院）；School of Statistics and Data Science, Nankai University（南开大学统计与数据科学学院）；Ant Tiansuan Security Lab, Ant Group（蚂蚁集团蚂蚁天算安全实验室）
