Graham Neubig

MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking

Sathyanarayanan Ramamoorthy, Vishwa Shah, Simran Khanuja, Zaid Sheikh, Shan Jie, Ann Chia, Shearman Chua, Graham Neubig

机构 * Carnegie Mellon University（卡内基梅隆大学） ； Defence Science and Technology Agency（国防科学与技术局） 纠错

Devstral: Fine-tuning Language Models for Coding Agent Applications

Abhinav Rastogi, Adam Yang, Albert Q. Jiang, Alexander H. Liu, Alexandre Sablayrolles, Amélie Héliou, Amélie Martin, Anmol Agarwal, Andy Ehrenberg, Andy Lo, Antoine Roux, Arthur Darcet, Arthur Mensch, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Clémence Lanfranchi, Clément Denoix, Corentin Barreau, Darius Dabert Devon Mizelle, Diego de las Casas, Elliot Chane-Sane, Emilien Fugier, Emma Bou Hanna, Gabrielle Berrada, Gauthier Delerce, Gauthier Guinet, Georgii Novikov, Graham Neubig, Guillaume Lample, Guillaume Martin, Himanshu Jaju, Jan Ludziejewski, Jason Rute, Jean-Malo Delignon, JeanHadrien Chabran, Joachim Studnia, Joep Barmentlo, Jonas Amar, Josselin Somerville Roberts, Julien Denize, Karan Saxena, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Kush Jain, Lélio Renard Lavaud, Léonard Blier, Lingxiao Zhao, Louis Martin, Lucile Saulnier, Luyu Gao, Marie Pellat, Mathilde Guillaumin, Mathis Felardos, Matthieu Dinot, Maxime Darrin, Maximilian Augustin, Mickaël Seznec, Neha Gupta, Nikhil Raghuraman, Olivier Duchenne, Patricia Wang, Patrick von Platen, Patryk Saffer, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Philomène Chagniot, Pierre Stock, Pravesh Agrawal, Rémi Delacourt, Roman Soletskyi, Romain Sauvestre, Sagar Vaze, Sanchit Gandhi, Sandeep Subramanian, Shashwat Dalal, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Teven Le Scao, Thibaut Lavril, Thibault Schueller, Thomas Foubert, Thomas Robert, Thomas Wang, Timothée Lacroix, Tom Bewley, Valeriia Nemychnikova, Victor Paltz, Virgile Richard, Wen-Ding Li, William Marshall, Xingyao Wang, Xuanyu Zhang, Yihan Wan, Yunhao Tang

机构 * Mistral AI 纠错

Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows

Valerie Chen, Ameet Talwalkar, Robert Brennan, Graham Neubig

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, Graham Neubig

机构 * Carnegie Mellon University（卡内基梅隆大学） ； Independent（独立研究者） ； Duke University（杜克大学） 纠错

Comments Preprint

Repetition Improves Language Model Embeddings

Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, Aditi Raghunathan

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

Comments ICLR 2025

Evaluating Language Models as Synthetic Data Generators

Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig

机构 * Carnegie Mellon University（卡内基梅隆大学） ； KAIST AI（韩国科学技术院人工智能研究所） ； University of Washington（华盛顿大学） ； NEC Laboratories Europe（NEC欧洲实验室） ； Ss. Cyril and Methodius University of Skopje（斯科普里塞奥尼和方法ius大学） 纠错

Comments ACL 2025 (main)

Inducing Programmatic Skills for Agentic Tasks

Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, Daniel Fried

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

Grounding Multilingual Multimodal LLMs With Cultural Knowledge

Jean de Dieu Nyandwi, Yueqi Song, Simran Khanuja, Graham Neubig

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering

Patrick Fernandes, Sweta Agrawal, Emmanouil Zaranis, André F. T. Martins, Graham Neubig

机构 * Carnegie Mellon University（卡内基梅隆大学） ； Instituto de Telecomunicações（电信研究所） ； Instituto Superior Técnico, Universidade de Lisboa（里斯本大学技术学院） ； Unbabel 纠错

RAGGED: Towards Informed Design of Scalable and Stable RAG Systems

Jennifer Hsia, Afreen Shaikh, Zhiruo Wang, Graham Neubig

机构 * Machine Learning Department, Carnegie Mellon University, Pittsburgh, USA（卡内基梅隆大学机器学习系） ； The Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA（卡内基梅隆大学语言技术研究所） 纠错

Comments Project page: https://github.com/neulab/ragged

Journal ref ICML 2025

AutoPresent: Designing Structured Visuals from Scratch

Jiaxin Ge, Zora Zhiruo Wang, Xuhui Zhou, Yi-Hao Peng, Sanjay Subramanian, Qinyue Tan, Maarten Sap, Alane Suhr, Daniel Fried, Graham Neubig, Trevor Darrell

机构 * University of California, Berkeley（加州大学伯克利分校） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

Beyond Browsing: API-Based Web Agents

Yueqi Song, Frank Xu, Shuyan Zhou, Graham Neubig

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

Comments 20 pages, 8 figures

NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples

Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, Deva Ramanan

机构 * Carnegie Mellon University（卡内基梅隆大学） ； University of Washington（华盛顿大学） 纠错

Comments Accepted to NeurIPS 24; We open-source our dataset at: https://huggingface.co/datasets/BaiqiL/NaturalBench ; Project page at: https://linzhiqiu.github.io/papers/naturalbench/

BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models

Lindia Tjuatja, Graham Neubig

Comments Accepted to ACL 2025 Main Conference

Training Software Engineering Agents and Verifiers with SWE-Gym

Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang

Comments Accepted at ICML 2025. Code at https://github.com/SWE-Gym/SWE-Gym

MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, Xiang Yue

机构 * Carnegie Mellon University（卡内基梅隆大学） ； M-A-P ； Nanyang Technological University（南洋理工大学） ； University of Waterloo（滑铁卢大学） ； The University of Manchester（曼彻斯特大学） 纠错

Comments ACL 2025 Main

Coding Agents with Multimodal Browsing are Generalist Problem Solvers

Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, Graham Neubig

What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length

Lindia Tjuatja, Graham Neubig, Tal Linzen, Sophie Hao

机构 * Carnegie Mellon University（卡内基梅隆大学） ； New York University（纽约大学） 纠错

Comments Accepted to NAACL 2025 (Main Conference)

Stereotype or Personalization? User Identity Biases Chatbot Recommendations

Anjali Kantharuban, Jeremiah Milbauer, Maarten Sap, Emma Strubell, Graham Neubig

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

A Gold Standard Dataset for the Reviewer Assignment Problem

Ivan Stelmakh, John Wieting, Sarina Xi, Graham Neubig, Nihar B. Shah

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham Neubig

Comments ACL 2025 Main

The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think

Seongyun Lee, Seungone Kim, Minju Seo, Yongrae Jo, Dongyoung Go, Hyeonbin Hwang, Jinho Park, Xiang Yue, Sean Welleck, Graham Neubig, Moontae Lee, Minjoon Seo

Comments Work in progress

VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge

Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, Xiang Yue

Comments 56 pages, 43 figures

OpenHands: An Open Platform for AI Software Developers as Generalist Agents

Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, Graham Neubig

机构 * UIUC（伊利诺伊大学香槟分校） ； CMU（卡内基梅隆大学） ； Yale（耶鲁大学） ； UC Berkeley（加州大学伯克利分校） ； Contextual AI（上下文AI） ； KAUST（科威特大学） ； ANU（澳大利亚国立大学） ； HCMUT（胡志明市理工大学） ； Alibaba（阿里巴巴） ； All Hands AI（所有手AI） 纠错

Comments Accepted by ICLR 2025; Code: https://github.com/All-Hands-AI/OpenHands

SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills

Boyuan Zheng, Michael Y. Fatemi, Xiaolong Jin, Zora Zhiruo Wang, Apurva Gandhi, Yueqi Song, Yu Gu, Jayanth Srinivasa, Gaowen Liu, Graham Neubig, Yu Su

Overtrained Language Models Are Harder to Fine-Tune

Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, Aditi Raghunathan

Comments 72 pages, 65 figures, 6 tables

The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models

Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo

Comments NAACL 2025 (Main Conference)

Towards Automatic Evaluation for Image Transcreation

Simran Khanuja, Vivek Iyer, Claire He, Graham Neubig

Comments To be presented at NAACL 2025

JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation

Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Kazuki Egashira, Jeonghun Baek, Xiang Yue, Graham Neubig, Kiyoharu Aizawa

Comments Accepted at NAACL 2025. Project page: https://mmmu-japanese-benchmark.github.io/JMMMU/

Benchmarking Failures in Tool-Augmented Language Models

Eduardo Treviño, Hugo Contant, James Ngai, Graham Neubig, Zora Zhiruo Wang

Journal ref 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics