大模型推理能力

Rationale-Aware Answer Verification by Pairwise Self-Evaluation

Akira Kawabata, Saku Sugawara

专题命中 推理评测 ：reasoning(abstract)；verifier(abstract)；分类 cs.CL

Comments EMNLP 2024

Are LLMs Good Zero-Shot Fallacy Classifiers?

Fengjun Pan, Xiaobao Wu, Zongrui Li, Anh Tuan Luu

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted to EMNLP2024 main conference

Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study

Gonzalo Martínez, José Alberto Hernández, Javier Conde, Pedro Reviriego, Elena Merino

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Journal ref ACM Transactions on Intelligent Systems and Technology, 2024

Pragmatic Competence Evaluation of Large Language Models for the Korean Language

Dojun Park, Jiwoo Lee, Hyeyun Jeong, Seohyun Park, Sungeun Lee

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments 38th Pacific Asia Conference on Language, Information and Computation

Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning

Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, Shiwei Liu

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments EMNLP 2024

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, Hinrich Schütze

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments EMNLP 2024 - Findings

NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian

Peng Liu, Lemei Zhang, Terje Farup, Even W. Lauvrak, Jon Espen Ingvaldsen, Simen Eide, Jon Atle Gulla, Zhirong Yang

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted at EMNLP 2024 Main Conference. Code available at https://github.com/Smartmedia-AI/NorGLM/

Task-oriented Prompt Enhancement via Script Generation

Chung-Yu Wang, Alireza DaghighFarsoodeh, Hung Viet Pham

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Comments 17 pages + reference

WebQuest: A Benchmark for Multimodal QA on Web Page Sequences

Maria Wang, Srinivas Sunkara, Gilles Baechler, Jason Lin, Yun Zhu, Fedir Zubach, Lei Shu, Jindong Chen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Can we only use guideline instead of shot in prompt?

Jiaxiang Chen, Song Wang, Zhucong Li, Wayne Xiong, Lizhen Qu, Zenglin Xu, Yuan Qi

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.AI

CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization

Ziwei Gong, Lin Ai, Harshsaiprasad Deshpande, Alexander Johnson, Emmy Phung, Zehui Wu, Ahmad Emami, Julia Hirschberg

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, Zack Hui

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

MAGDA: Multi-agent guideline-driven diagnostic assistance

David Bani-Harouni, Nassir Navab, Matthias Keicher

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Models

Xiyuan Zhou, Huan Zhao, Yuheng Cheng, Yuji Cao, Gaoqi Liang, Guolong Liu, Wenxuan Liu, Yan Xu, Junhua Zhao

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.AI

IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, Rifat Shahriyar

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

IAI Group at CheckThat! 2024: Transformer Models and Data Augmentation for Checkworthy Claim Detection

Peter Røysland Aarnes, Vinay Setty, Petra Galuščáková

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Comments Accepted to CLEF2024 CheckThat!

Revisiting Relation Extraction in the era of Large Language Models

Somin Wadhwa, Silvio Amir, Byron C. Wallace

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments Accepted to ACL 2023

Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping

Wenhao Zhu, Sizhe Liu, Shujian Huang, Shuaijie She, Chris Wendler, Jiajun Chen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

Towards Temporal Change Explanations from Bi-Temporal Satellite Images

Ryo Tsujimoto, Hiroki Ouchi, Hidetaka Kamigaito, Taro Watanabe

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

Comments 11 pages, 6 figures

Leveraging large language models for nano synthesis mechanism explanation: solid foundations or mere conjectures?

Yingming Pu, Liping Huang, Tao Lin, Hongyu Chen

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.LG

Sports Intelligence: Assessing the Sports Understanding Capabilities of Language Models through Question Answering from Text to Video

Zhengbang Yang, Haotian Xia, Jingxi Li, Zezhi Chen, Zhuangdi Zhu, Weining Shen

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.CL

IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models

Qiyao Wang, Jianguo Huang, Shule Lu, Yuan Lin, Kan Xu, Liang Yang, Hongfei Lin

专题命中 推理评测 ：reasoning(abstract)；CoT(abstract)；分类 cs.CL

Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach

Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, Jun Wang

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

LogiCode: an LLM-Driven Framework for Logical Anomaly Detection

Yiheng Zhang, Yunkang Cao, Xiaohao Xu, Weiming Shen

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.LG

A Survey of Useful LLM Evaluation

Ji-Lun Peng, Sijia Cheng, Egil Diau, Yung-Yu Shih, Po-Heng Chen, Yen-Ting Lin, Yun-Nung Chen

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.CL

CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua

专题命中 推理评测 ：chain-of-thought(abstract)；CoT(abstract)；分类 cs.CL

Comments Accepted to ACL 2024. Camera Ready. Our dataset is available at https://github.com/zt991211/CLAMBER

WorldAfford: Affordance Grounding based on Natural Language Instructions

Changmao Chen, Yuren Cong, Zhen Kan

专题命中 推理评测 ：reasoning(abstract)；chain-of-thought(abstract)；分类 cs.AI

Evaluating and Modeling Social Intelligence: A Comparative Study of Human and AI Capabilities

Junqi Wang, Chunhui Zhang, Jiapeng Li, Yuxi Ma, Lixing Niu, Jiaheng Han, Yujia Peng, Yixin Zhu, Lifeng Fan

专题命中 推理评测 ：reasoning(abstract)；planning(abstract)；分类 cs.AI

Comments Also published in Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci), 2024

SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications

Liang Xu, Lei Zhu, Yaotong Wu, Hang Xue

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL

Comments 11 pages, 19 figures, and tables

Language Models Still Struggle to Zero-shot Reason about Time Series

Mike A. Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, Tim Althoff

专题命中 推理评测 ：reasoning(abstract)；logical reasoning(abstract)；分类 cs.CL