大模型推理能力

Large Language Models in the Clinic: A Comprehensive Benchmark

Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang, Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam, Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur, Lei Clifton, David A. Clifton

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted at EMNLP 2024 Main Conference

Athena: Retrieval-augmented Legal Judgment Prediction with Large Language Models

Xiao Peng, Liang Chen

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments 13 pages, 6 figures

Declarative Knowledge Distillation from Large Language Models for Visual Question Answering Datasets

Thomas Eiter, Jan Hadl, Nelson Higuera, Johannes Oetsch

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Presented at NeLaMKRR@KR, 2024 (arXiv:2410.05339)

Generative Design of Periodic Orbits in the Restricted Three-Body Problem

Alvaro Francisco Gil, Walther Litteri, Victor Rodriguez-Fernandez, David Camacho, Massimiliano Vasile

专题命中 推理评测 ：planning(abstract)；分类 cs.AI、cs.LG

Comments SPAICE Conference 2024 (7 pages)

Journal ref Proceedings of SPAICE 2024, pp. 430-436

Extended Japanese Commonsense Morality Dataset with Masked Token and Label Enhancement

Takumi Ohashi, Tsubasa Nakagawa, Hitoshi Iyatomi

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Journal ref Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM '24), October 21--25, 2024, Boise, ID, USA

Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models

Yeeun Kim, Young Rok Choi, Eunkyung Choi, Jinhwan Choi, Hai Jin Park, Wonseok Hwang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 Findings

DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models

Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, Kang Liu

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024

OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting

Xukai Liu, Ye Liu, Kai Zhang, Kehang Wang, Qi Liu, Enhong Chen

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2024 Main

LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

Yuan Chiang, Elvis Hsieh, Chia-Hong Chou, Janosh Riebesell

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 32 pages, 5 figures

Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles

Qi Chen, Bowen Zhang, Gang Wang, Qi Wu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted by NeurIPS 2024

Probing the Robustness of Theory of Mind in Large Language Models

Christian Nickel, Laura Schrewe, Lucie Flek

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models

Dahyun Kim, Sukyung Lee, Yungi Kim, Attapol Rutherford, Chanjun Park

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, Peter Clark

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NeurIPS 2024 (Benchmark Track, Spotlight)

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, Gongshen Liu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP Findings 2024

Why Would You Suggest That? Human Trust in Language Model Responses

Manasi Sharma, Ho Chit Siu, Rohan Paleja, Jaime D. Peña

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Journal ref ICML Humans, Algorithmic Decision-Making and Society: Modeling Interactions and Impact Workshop 2024

NL-Eye: Abductive NLI for Images

Mor Ventura, Michael Toker, Nitay Calderon, Zorik Gekhman, Yonatan Bitton, Roi Reichart

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 Main. We release our code and data publicly at https://github.com/MozerWang/Loong

Automated Red Teaming with GOAT: the Generative Offensive Agent Tester

Maya Pavlova, Erik Brinkman, Krithika Iyer, Vitor Albiero, Joanna Bitton, Hailey Nguyen, Joe Li, Cristian Canton Ferrer, Ivan Evtimov, Aaron Grattafiori

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

FABLES: Evaluating faithfulness and content selection in book-length summarization

Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, Mohit Iyyer

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments preprint - 39 pages

Journal ref 1st Conference on Language Modeling (COLM 2024)

Cause and Effect: Can Large Language Models Truly Understand Causality?

Swagata Ashwani, Kshiteesh Hegde, Nishith Reddy Mannuru, Mayank Jindal, Dushyant Singh Sengar, Krishna Chaitanya Rao Kathala, Dishant Banga, Vinija Jain, Aman Chadha

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments AI Trustworthiness and Risk Assessment for Challenged Contexts (ATRACC) AAAI 2024 Fall Symposium

Integrating Hierarchical Semantic into Iterative Generation Model for Entailment Tree Explanation

Qin Wang, Jianzhou Feng, Yiming Xu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

AXCEL: Automated eXplainable Consistency Evaluation using LLMs

P Aditya Sreekar, Sahil Verma, Suransh Chopra, Sarik Ghazarian, Abhishek Persad, Narayanan Sadagopan

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Prompting Large Language Models for Supporting the Differential Diagnosis of Anemia

Elisa Castagnari, Lillian Muyama, Adrien Coulet

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Journal ref LLMs4MI 2024 @FLLM 2024, IEEE, Nov 2024, Dubai, United Arab Emirates

A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review

Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, Piyush Mathur, Giovanni E. Cacciamani, Cong Sun, Yifan Peng, Yanshan Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation

Juhwan Choi, Jungmin Yun, Kyohoon Jin, YoungBin Kim

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024: Camera-ready version

Behavioral Bias of Vision-Language Models: A Behavioral Finance View

Yuhang Xiao, Yudi Lin, Ming-Chang Chiu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments ICML 2024 Workshop on Large Language Models and Cognition

A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations

Jinqiang Wang, Huansheng Ning, Yi Peng, Qikai Wei, Daniel Tesfai, Wenwei Mao, Tao Zhu, Runhe Huang

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

Comments 25 pages,4 figures

XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs

Zichen Chen, Jianda Chen, Ambuj Singh, Misha Sra

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted at EMNLP 2024 Main. The dataset and code are available at https://github.com/chen-zichen/XplainLLM_dataset.git

Towards Faithful Knowledge Graph Explanation Through Deep Alignment in Commonsense Question Answering

Weihe Zhai, Arkaitz Zubiaga

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 Main

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries

Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij, Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Firat, Angeliki Lazaridou, Jean-Baptiste Lespiau, Nithya Attaluri, Kate Olszewska

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG