大模型推理能力

MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents

Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, Jonathan H. Chen

专题命中 推理评测 ：planning(abstract)；分类 cs.AI、cs.LG

WHODUNIT: Evaluation benchmark for culprit detection in mystery stories

Kshitij Gupta

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Forbidden Science: Dual-Use AI Challenge Benchmark and Scientific Refusal Tests

David Noever, Forrest McKee

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

LM2: Large Memory Models

Jikun Kang, Wenqi Wu, Filippos Christianos, Alex J. Chan, Fraser Greenlee, George Thomas, Marvin Purtorab, Andy Toulis

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

CSEval: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLMs

Amey Hengle, Aswini Kumar, Anil Bandhakavi, Tanmoy Chakraborty

专题命中 推理评测 ：CoT(abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 5 figures

Entity Alignment with Noisy Annotations from Large Language Models

Shengyuan Chen, Qinggang Zhang, Junnan Dong, Wen Hua, Qing Li, Xiao Huang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Journal ref NeurIPS 2024

MEETING DELEGATE: Benchmarking LLMs on Attending Meetings on Our Behalf

Lingxiang Hu, Shurun Yuan, Xiaoting Qin, Jue Zhang, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels

Malte Mosbach, Jan Niklas Ewertz, Angel Villar-Corrales, Sven Behnke

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Are Language Models Up to Sequential Optimization Problems? From Evaluation to a Hegelian-Inspired Enhancement

Soheil Abbasloo

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

The role of positional encodings in the ARC benchmark

Guilherme H. Bandeira Costa, Miguel Freire, Arlindo L. Oliveira

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Synthetic User Behavior Sequence Generation with Large Language Models for Smart Homes

Zhiyao Xu, Dan Zhao, Qingsong Zou, Jingyu Xiao, Yong Jiang, Zhenhui Yuan, Qing Li

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Beyond the Veil of Similarity: Quantifying Semantic Continuity in Explainable AI

Qi Huang, Emanuele Mezzi, Osman Mutlu, Miltiadis Kofinas, Vidya Prasad, Shadnan Azwad Khan, Elena Ranguelova, Niki van Stein

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments 25 pages, accepted at the world conference of explainable AI, 2024, Malta

FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Data

Deren Lei, Yaxi Li, Siyao Li, Mengya Hu, Rui Xu, Ken Archer, Mingyu Wang, Emily Ching, Alex Deng

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments NAACL 2025

Histoires Morales: A French Dataset for Assessing Moral Alignment

Thibaud Leteno, Irina Proskurina, Antoine Gourru, Julien Velcin, Charlotte Laclau, Guillaume Metzler, Christophe Gravier

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NAACL 2025

Graph Transformers for inverse physics: reconstructing flows around arbitrary 2D airfoils

Gregory Duthé, Imad Abdallah, Eleni Chatzi

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding

Sankalp KJ, Ashutosh Kumar, Laxmaan Balaji, Nikunj Kotecha, Vinija Jain, Aman Chadha, Sreyoshi Bhaduri

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Zep: A Temporal Knowledge Graph Architecture for Agent Memory

Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, Daniel Chalef

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 12 pages, 3 tables

Can OpenAI o1 Reason Well in Ophthalmology? A 6,990-Question Head-to-Head Evaluation Study

Sahana Srinivasan, Xuguang Ai, Minjie Zou, Ke Zou, Hyunjae Kim, Thaddaeus Wai Soon Lo, Krithi Pushpanathan, Yiming Kong, Anran Li, Maxwell Singer, Kai Jin, Fares Antaki, David Ziyou Chen, Dianbo Liu, Ron A. Adelman, Qingyu Chen, Yih Chung Tham

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 44 pages

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Tadesse Kebede Guge, Tombekai Vangoni Sherman, Pontus Stenetorp

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NAACL 2025 (main conference)

SRMT: Shared Memory for Multi-agent Lifelong Pathfinding

Alsu Sagirova, Yuri Kuratov, Mikhail Burtsev

专题命中 推理评测 ：planning(abstract)；分类 cs.AI、cs.LG

Comments 16 pages, 11 figures

MMVU: Measuring Expert-Level Multi-Discipline Video Understanding

Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, Zhijian Xu, Chengye Wang, Weifeng Pan, Ziyao Shangguan, Xiangru Tang, Zhenwen Liang, Yixin Liu, Chen Zhao, Arman Cohan

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

A Collection of Question Answering Datasets for Norwegian

Vladislav Mikhailov, Petter Mæhlum, Victoria Ovedie Chruickshank Langø, Erik Velldal, Lilja Øvrelid

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted for NoDaLiDa / Baltic-HLT 2025

Clinical trial cohort selection using Large Language Models on n2c2 Challenges

Chi-en Amy Tai, Xavier Tannier

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models

Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, Xin Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos

Mehmet Saygin Seyfioglu, Wisdom O. Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, Linda Shapiro

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

SALT: Sales Autocompletion Linked Business Tables Dataset

Tassilo Klein, Clemens Biehl, Margarida Costa, Andre Sres, Jonas Kolk, Johannes Hoffart

专题命中 推理评测 ：planning(abstract)；分类 cs.AI、cs.LG

Comments Table Representation Learning Workshop at NeurIPS 2024

AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, Mariette Awad

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset

Chongjian Yue, Xinrun Xu, Xiaojun Ma, Lun Du, Zhiming Ding, Shi Han, Dongmei Zhang, Qi Zhang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments ICASSP 2025

Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets

Vatsal Gupta, Pranshu Pandya, Tushar Kataria, Vivek Gupta, Dan Roth

专题命中 推理评测 ：CoT(abstract)；分类 cs.CL、cs.AI

Comments 23 pages, 16 Figure, 10 Tables

Benchmarking Large Language Model Uncertainty for Prompt Optimization

Pei-Fu Guo, Yun-Da Tsai, Shou-De Lin

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG