大模型推理能力

CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization

Brihi Joshi, Sriram Venkatapathy, Mohit Bansal, Nanyun Peng, Haw-Shiuan Chang

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

Bias Testing and Mitigation in LLM-based Code Generation

Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, Heming Cui

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.AI

Comments Accepted by TOSEM

Goal2Story: A Multi-Agent Fleet based on Privately Enabled sLLMs for Impacting Mapping on Requirements Elicitation

Xinkai Zou, Yan Liu, Xiongbo Shi, Chen Yang

专题命中 推理评测 ：CoT(abstract)；planning(abstract)；分类 cs.AI

CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models

Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, Libo Qin

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Comments Accepted at AAAI 2025; Project Page: https://github.com/czhhzc/CoMT

RocketEval: Efficient Automated LLM Evaluation via Grading Checklist

Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, Jianghong Ma

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted by ICLR 2025: https://openreview.net/forum?id=zJjzNj6QUe

Psy-Insight: Explainable Multi-turn Bilingual Dataset for Mental Health Counseling

Keqi Chen, Zekai Sun, Yuhua Wen, Huijun Lian, Yingming Gao, Ya Li

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis

Jiahao Gai, Hao Mark Chen, Zhican Wang, Hongyu Zhou, Wanru Zhao, Nicholas Lane, Hongxiang Fan

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.LG

Comments Paper accepted by ASP-DAC'25

A Survey on LLM-powered Agents for Recommender Systems

Qiyao Peng, Hongtao Liu, Hua Huang, Qing Yang, Minglai Shao

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

CodEv: An Automated Grading Framework Leveraging Large Language Models for Consistent and Constructive Feedback

En-Qi Tseng, Pei-Cing Huang, Chan Hsu, Peng-Yi Wu, Chan-Tung Ku, Yihuang Kang

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.AI

Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents

Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Caiming Xiong, Shiva Kumar Pentyala, Chien-Sheng Wu

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models

Zhongzhan Huang, Shanshan Zhong, Pan Zhou, Shanghua Gao, Marinka Zitnik, Liang Lin

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

Comments Accepted by TPAMI. arXiv admin note: text overlap with arXiv:2312.02439

How to Get Your LLM to Generate Challenging Problems for Evaluation

Arkil Patel, Siva Reddy, Dzmitry Bahdanau

专题命中 推理评测 ：reasoning(abstract)；math reasoning(abstract)；分类 cs.CL

Benchmarking Language Model Creativity: A Case Study on Code Generation

Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, Sanjeev Khudanpur, Meng Jiang, Daniel Khashabi

专题命中 推理评测 ：reasoning(abstract)；self-correction(abstract)；分类 cs.CL

ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

Haiyang Shen, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, Yun Ma

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

Comments ICLR'25: https://openreview.net/forum?id=kKILfPkhSz

OnionEval: An Unified Evaluation of Fact-conflicting Hallucination for Small-Large Language Models

Chongren Sun, Yuran Li, Di Wu, Benoit Boulet

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans

Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, Raymond Mooney

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2024 Main Conference

OpenLKA: an open dataset of lane keeping assist from market autonomous vehicles

Yuhang Wang, Abdulaziz Alhuraish, Shengming Yuan, Shuyi Wang, Hao Zhou

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.LG

SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity

Pengfei Jing, Mengyun Tang, Xiaorong Shi, Xing Zheng, Sen Nie, Shi Wu, Yong Yang, Xiapu Luo

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension

Yanbo Fang, Ruixiang Tang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

How susceptible are LLMs to Logical Fallacies?

Amirreza Payandeh, Dan Pluth, Jordan Hosier, Xuesu Xiao, Vijay K. Gurbani

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Journal ref https://aclanthology.org/2024.lrec-main.726/

Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios

Zhi Chen, Lingxiao Jiang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

Comments Paper accepted to the SANER 2025 Conference Research Track

From Models to Microtheories: Distilling a Model's Topical Knowledge for Grounded Question Answering

Nathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Sam Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations

Sunjun Kweon, Byungjin Choi, Gyouk Chu, Junyeong Song, Daeun Hyeon, Sujin Gan, Jueon Kim, Minkyu Kim, Rae Woong Park, Edward Choi

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models

Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, Liwen Zhang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data

Junhao Liu, Siwei Xu, Lei Zhang, Jing Zhang

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Leveraging Large Language Models for Comparative Literature Summarization with Reflective Incremental Mechanisms

Fernando Gabriela Garcia, Spencer Burns, Harrison Fuller

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

Comments 8 pages

Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos

Zhimin Shao, Jialang Xu, Danail Stoyanov, Evangelos B. Mazomenos, Yueming Jin

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Comments 8 pages, 4 figures

Journal ref IEEE Robotics and Automation Letters, vol. 9, no. 12, pp. 11513-11520, Dec. 2024

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, Jiashi Feng

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

Comments EMNLP 2024

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, Wenhu Chen

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

Comments This version has been accepted and published at NeurIPS 2024 Track Datasets and Benchmarks (Spotlight)

MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering

Yizhen Luo, Zikun Nie, Massimo Hong, Suyuan Zhao, Hao Zhou, Zaiqing Nie

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.LG

Comments NeurIPS 2024 poster