大模型推理能力

Measuring Taiwanese Mandarin Language Understanding

Po-Heng Chen, Sijia Cheng, Wei-Lin Chen, Yen-Ting Lin, Yun-Nung Chen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Preprint. Under review

Can a GPT4-Powered AI Agent Be a Good Enough Performance Attribution Analyst?

Bruno de Melo, Jamiel Sheikh

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.AI

HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models

Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jaecheol Lee, Je Won Yeom, Jihyu Jung, Jung Woo Kim, Songseong Kim

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments Accepted at LREC-COLING 2024

Metaphor Understanding Challenge Dataset for LLMs

Xiaoyu Tong, Rochelle Choenni, Martha Lewis, Ekaterina Shutova

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles

Shulin Huang, Shirong Ma, Yinghui Li, Mengzuo Huang, Wuhe Zou, Weidong Zhang, Hai-Tao Zheng

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments Accepted by LREC-COLING 2024

Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games

Dekun Wu, Haochen Shi, Zhiyuan Sun, Bang Liu

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments

Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang, Wei-Wei Tu, Zhaofeng He, Lijie Wen

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models

Ayo Adedeji, Sarita Joshi, Brendan Doohan

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments 31 pages, 17 figures

Assertion Detection Large Language Model In-context Learning LoRA Fine-tuning

Yuelyu Ji, Zeshui Yu, Yanshan Wang

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

OpenPI2.0: An Improved Dataset for Entity Tracking in Texts

Li Zhang, Hainiu Xu, Abhinav Kommula, Chris Callison-Burch, Niket Tandon

专题命中 推理评测 ：chain-of-thought(abstract)；planning(abstract)；分类 cs.CL

Comments In EACL 2024

CMMLU: Measuring massive multitask language understanding in Chinese

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, Timothy Baldwin

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, Feng Zhao

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

Comments Project: https://open-compass.github.io/T-Eval

Speech Translation with Large Language Models: An Industrial Practice

Zhichao Huang, Rong Ye, Tom Ko, Qianqian Dong, Shanbo Cheng, Mingxuan Wang, Hang Li

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments Technical report. 13 pages. Demo: https://speechtranslation.github.io/llm-st/

FELM: Benchmarking Factuality Evaluation of Large Language Models

Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, Junxian He

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted by NeurIPS 2023 Track on Datasets and Benchmarks

FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models

Yimin Jing, Renren Jin, Jiahao Hu, Huishi Qiu, Xiaohua Wang, Peng Wang, Deyi Xiong

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments Work in progress

CompCodeVet: A Compiler-guided Validation and Enhancement Approach for Code Dataset

Le Chen, Arijit Bhattacharjee, Nesreen K. Ahmed, Niranjan Hasabnis, Gal Oren, Bin Lei, Ali Jannesari

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.LG

LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation

Shengqiang Zhang, Philipp Wicke, Lütfi Kerem Şenel, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, Barbara Plank, Hinrich Schütze

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

Comments 6 pages, 4 figures. The video and code of LoHoRavens are available at https://cisnlp.github.io/lohoravens-webpage/

Answering Unseen Questions With Smaller Language Models Using Rationale Generation and Dense Retrieval

Tim Hartill, Diana Benavides-Prado, Michael Witbrock, Patricia J. Riddle

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Extractive Summarization via ChatGPT for Faithful Summary Generation

Haopeng Zhang, Xiao Liu, Jiawei Zhang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Findings of EMNLP 2023

Guiding Large Language Models via Directional Stimulus Prompting

Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, Xifeng Yan

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted by NeurIPS2023. The code and data are available at https://github.com/Leezekun/Directional-Stimulus-Prompting

A Closer Look into Automatic Evaluation Using Large Language Models

Cheng-Han Chiang, Hung-yi Lee

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments EMNLP 2023 findings (short paper). Code: https://github.com/d223302/A-Closer-Look-To-LLM-Evaluation/

Aligning Large Language Models for Clinical Tasks

Supun Manathunga, Isuru Hettigoda

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments 10 pages, 3 figures

Solving and Generating NPR Sunday Puzzles with Large Language Models

Jingmiao Zhao, Carolyn Jane Anderson

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments To appear in the Proceedings of the 14th International Conference on Computational Creativity (ICCC)

IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

Wenhao Yu, Meng Jiang, Peter Clark, Ashish Sabharwal

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation

Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation

Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, Yue Zhang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding

Sayan Ghosh, Shashank Srivastava

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments ACL 2022

Evaluation of LLMs for Process Model Analysis and Optimization

Akhil Kumar, Jianliang Leon Zhao, Om Dobariya

专题命中 推理评测 ：reasoning(abstract,comments)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages, 5 tables, 4 figures; full research paper currently under review for the Workshop on Information Technologies and Systems (WITS) 2025. The paper presents a comprehensive evaluation of large language models (LLMs) for business process model analysis and optimization, including error detection, reasoning, and scenario-based redesign

Rethinking Human Preference Evaluation of LLM Rationales

Ziang Li, Manasi Ganti, Zixian Ma, Helena Vasconcelos, Qijia He, Ranjay Krishna

机构 * University of Washington（华盛顿大学） ； Stanford University（斯坦福大学） 纠错

专题命中 推理评测 ：reasoning(abstract,comments)；分类 cs.CL、cs.AI；planning(comments)

Comments Published in the XLLM-Reason-Plan Workshop on the Application of LLM Explainability to Reasoning and Planning at COLM 2025

Journal ref Proceedings of the XLLM-Reason-Plan Workshop, Conference on Language Modeling (COLM) 2025