大模型推理能力

The Rosetta Paradox: Domain-Specific Performance Inversions in Large Language Models

Basab Jha, Ujjwal Puri

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 7 figures

Prompted Contextual Vectors for Spear-Phishing Detection

Daniel Nahmias, Gal Engelberg, Dan Klein, Asaf Shabtai

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

GPTEval: A Survey on Assessments of ChatGPT and GPT-4

Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, Erik Cambria

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Journal ref Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (2024) 7844-7866

Chumor 2.0: Towards Benchmarking Chinese Humor Understanding

Ruiqi He, Yushu He, Longju Bai, Jiarui Liu, Zhenjie Sun, Zenghao Tang, He Wang, Hanchen Xia, Rada Mihalcea, Naihao Deng

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments arXiv admin note: substantial text overlap with arXiv:2406.12754

GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking

Darshan Deshpande, Selvan Sunitha Ravi, Sky CH-Wang, Bartosz Mielczarek, Anand Kannappan, Rebecca Qian

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Memory-Augmented Agent Training for Business Document Understanding

Jiale Liu, Yifan Zeng, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, Qingyun Wu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 11 pages, 8 figures

Adapting to Non-Stationary Environments: Multi-Armed Bandit Enhanced Retrieval-Augmented Generation on Knowledge Graphs

Xiaqiang Tang, Jian Li, Nan Du, Sihong Xie

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments AAAI 2025

Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs

David Restrepo, Chenwei Wu, Zhengxu Tang, Zitao Shuai, Thao Nguyen Minh Phan, Jun-En Ding, Cong-Tinh Dao, Jack Gallifant, Robyn Gayle Dychiao, Jose Carlo Artiaga, André Hiroshi Bando, Carolina Pelegrini Barbosa Gracitelli, Vincenz Ferrer, Leo Anthony Celi, Danielle Bitterman, Michael G Morley, Luis Filipe Nakayama

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments Accepted at the AAAI 2025 Artificial Intelligence for Social Impact Track (AAAI-AISI 2025)

Cognition Chain for Explainable Psychological Stress Detection on Social Media

Xin Wang, Boyan Gao, Yi Dai, Lei Cao, Liang Zhao, Yibo Yang, David Clifton

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 26 pages, 12 figures, 6 tables

A NotSo Simple Way to Beat Simple Bench

Soham Sane, Angus McLean

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 29 pages, 11 Figures

Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models

Jie Chen, Xintian Han, Yu Ma, Xun Zhou, Liang Xiang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

IRR: Image Review Ranking Framework for Evaluating Vision-Language Models

Kazuki Hayashi, Kazuma Onishi, Toma Suzuki, Yusuke Ide, Seiji Gobara, Shigeki Saito, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 18pages, Accepted at COLING25

SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Tasks

Tianhao Li, Jingyu Lu, Chuangxin Chu, Tianyu Zeng, Yujia Zheng, Mei Li, Haotian Huang, Bin Wu, Zuoxian Liu, Kai Ma, Xuejing Yuan, Xingkai Wang, Keyan Ding, Huajun Chen, Qiang Zhang

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?

Zhikai Lei, Tianyi Liang, Hanglei Hu, Jin Zhang, Yunhua Zhou, Yunfan Shao, Linyang Li, Chenchui Li, Changbo Wang, Hang Yan, Qipeng Guo

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 13 figures

Piecing It All Together: Verifying Multi-Hop Multimodal Claims

Haoran Wang, Aman Rangapur, Xiongxiao Xu, Yueqing Liang, Haroon Gharwi, Carl Yang, Kai Shu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments COLING 2025

Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation

Javad Seraj, Mohammad Mahdi Mohajeri, Mohammad Javad Dousti, Majid Nili Ahmadabadi

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

LLM-BIP: Structured Pruning for Large Language Models with Block-Wise Forward Importance Propagation

Haihang Wu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, Caifeng Shan, Ran He

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Produced by MME+MMBench+LLaVA Teams. Project Page: https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Benchmarks

A Simple Model of Inference Scaling Laws

Noam Levi

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments 12 pages, 7 figures

Socio-Emotional Response Generation: A Human Evaluation Protocol for LLM-Based Conversational Systems

Lorraine Vanel, Ariel R. Ramos Vela, Alya Yacoubi, Chloé Clavel

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

Journal ref AHRI 2024, Sep 2024, Glasgow, United Kingdom

DeiSAM: Segment Anything with Deictic Prompting

Hikaru Shindo, Manuel Brack, Gopika Sudhakaran, Devendra Singh Dhami, Patrick Schramowski, Kristian Kersting

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at NeurIPS 2024

ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios

Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Tao Ji, Qi Zhang, Tao Gui, Xuanjing Huang

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

Comments Accepted by COLING 2025 conference

Are LLMs good pragmatic speakers?

Mingyue Jian, N. Siddharth

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

A Comprehensive Evaluation of Large Language Models on Aspect-Based Sentiment Analysis

Changzhi Zhou, Dandan Song, Yuhang Tian, Zhijing Wu, Hao Wang, Xinyu Zhang, Jun Yang, Ziyi Yang, Shuhao Zhang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem

Qianli Wang, Tatiana Anikina, Nils Feldhus, Simon Ostermann, Sebastian Möller, Vera Schmitt

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments Accepted at COLING 2025; long paper

GLaPE: Gold Label-agnostic Prompt Evaluation and Optimization for Large Language Model

Xuanchang Zhang, Zhuosheng Zhang, Hai Zhao

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments EMNLP 2024

ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges

Rao Fu, Ziyang Luo, Hongzhan Lin, Zhen Ye, Jing Ma

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Faithfulness and the Notion of Adversarial Sensitivity in NLP Explanations

Supriya Manna, Niladri Sett

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted as a Full Paper at EMNLP 2024 Workshop BlackBoxNLP

Journal ref Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2024

Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

Omkar Khade, Shruti Jagdale, Abhishek Phaltankar, Gauri Takalikar, Raviraj Joshi

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG