大模型推理能力

Understanding Social Reasoning in Language Models with Language Models

Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, Noah D. Goodman

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models

Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, Pascal Vincent

专题命中 推理评测 ：reasoning(title)；chain-of-thought(abstract)；分类 cs.CL、cs.AI

Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms

Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Jiwan Chung, Yejin Son, Yejin Choi, Youngjae Yu

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at EMNLP 2023 (long)

Evaluating the Effectiveness of Retrieval-Augmented Large Language Models in Scientific Document Reasoning

Sai Munikoti, Anurag Acharya, Sridevi Wagle, Sameera Horawalavithana

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments 5 pages

Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data

Mubashara Akhtar, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, Elena Simperl

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted at EMNLP 2023 (Findings)

ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense

Kankan Zhou, Eason Lai, Wei Bin Au Yeong, Kyriakos Mouratidis, Jing Jiang

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments This is the camera-ready version of the paper that will be published in the EMNLP 2023 Findings (Singapore, 6-10 December 2023)

HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models

Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yulong Chen, Naihao Deng

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted at Findings of EMNLP 2023

Journal ref Findings of EMNLP 2023

Reasoning about Ambiguous Definite Descriptions

Stefan F. Schouten, Peter Bloem, Ilia Markov, Piek Vossen

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2023 Findings

BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology

Odhran O'Donoghue, Aleksandar Shtedritski, John Ginger, Ralph Abboud, Ali Essa Ghareeb, Justin Booth, Samuel G Rodriques

专题命中 推理评测 ：planning(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2023. Dataset and code: https://github.com/bioplanner/bioplanner

How to Enhance Causal Discrimination of Utterances: A Case on Affective Reasoning

Hang Chen, Jing Luo, Xinyu Yang, Wenjing Zhu

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments accepted via EMNLP2023-main

NEWTON: Are Large Language Models Capable of Physical Reasoning?

Yi Ru Wang, Jiafei Duan, Dieter Fox, Siddhartha Srinivasa

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2023 Findings; 8 pages, 3 figures, 7 tables; Project page: https://newtonreasoning.github.io

MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models

Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, Kang Liu

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2023 Findings

Back to the Future: Towards Explainable Temporal Reasoning with Large Language Models

Chenhan Yuan, Qianqian Xie, Jimin Huang, Sophia Ananiadou

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments 14 pages, 5 figures, code and dataset: https://github.com/chenhan97/TimeLlama

GPT-Neo for commonsense reasoning -- a theoretical and practical lens

Rohan Kashyap, Vivek Kashyap, Narendra C. P.

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.LG

ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning

Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.LG

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models

Jaemin Cho, Abhay Zala, Mohit Bansal

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments ICCV 2023 (34 pages; see appendix for version changelog)

Counterfactual Reasoning for Bias Evaluation and Detection in a Fairness under Unawareness setting

Giandomenico Cornacchia, Vito Walter Anelli, Fedelucio Narducci, Azzurra Ragone, Eugenio Di Sciascio

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI、cs.LG

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments 143 pages, 79 tables, 4 figures

Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine

Thomas Savage, Ashwin Nayak, Robert Gallo, Ekanath Rangan, Jonathan H Chen

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

NovPhy: A Testbed for Physical Reasoning in Open-world Environments

Chathura Gamage, Vimukthini Pinto, Cheng Xue, Peng Zhang, Ekaterina Nikonova, Matthew Stephenson, Jochen Renz

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.AI、cs.LG

Comments Testbed website: https://github.com/phy-q/novphy

ARB: Advanced Reasoning Benchmark for Large Language Models

Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, Aran Komatsuzaki

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.LG

Comments Submitted to NeurIPS Datasets and Benchmarks Track

Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

Qingyu Tan, Hwee Tou Ng, Lidong Bing

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments ACL 2023

Knowledge Graph-Augmented Korean Generative Commonsense Reasoning

Dahyun Jung, Jaehyung Seo, Jaewook Lee, Chanjun Park, Heuiseok Lim

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted for Data-centric Machine Learning Research (DMLR) Workshop at ICML 2023

Emotion Flip Reasoning in Multiparty Conversations

Shivani Kumar, Shubham Dudeja, Md Shad Akhtar, Tanmoy Chakraborty

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments Paper accepted in IEEE Transaction on AI. 12 pages, 5 figures, 11 tables

Relational Temporal Graph Reasoning for Dual-task Dialogue Language Understanding

Bowen Xing, Ivor W. Tsang

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE TPAMI). arXiv admin note: substantial text overlap with arXiv:2203.03856

Multi-Task Training with In-Domain Language Models for Diagnostic Reasoning

Brihat Sharma, Yanjun Gao, Timothy Miller, Matthew M. Churpek, Majid Afshar, Dmitriy Dligach

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted to the Proceedings of the 5th Clinical NLP Workshop at ACL

Evaluating the Capabilities of Multi-modal Reasoning Models with Synthetic Task Data

Nathan Vaska, Victoria Helus

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.LG

ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models

Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, Dongkuan Xu

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions

Mo Yu, Yi Gu, Xiaoxiao Guo, Yufei Feng, Xiaodan Zhu, Michael Greenspan, Murray Campbell, Chuang Gan

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI

Comments arXiv admin note: text overlap with arXiv:2010.09788

Semantic-aware Dynamic Retrospective-Prospective Reasoning for Event-level Video Question Answering

Chenyang Lyu, Tianbo Ji, Yvette Graham, Jennifer Foster

专题命中 推理评测 ：reasoning(title,abstract)；分类 cs.CL、cs.AI