大模型推理能力

Knowledge Graphs are not Created Equal: Exploring the Properties and Structure of Real KGs

Nedelina Teneva, Estevam Hruschka

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments Accepted at NeurIPS 2023 New Frontiers in Graph Learning Workshop (NeurIPS GLFrontiers 2023)

TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs

Shuyi Xie, Wenlin Yao, Yong Dai, Shaobo Wang, Donlin Zhou, Lifeng Jin, Xinhua Feng, Pengzhi Wei, Yujie Lin, Zhichao Hu, Dong Yu, Zhengyou Zhang, Jing Nie, Yuhong Liu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

CRAB: Assessing the Strength of Causal Relationships Between Real-world Events

Angelika Romanou, Syrielle Montariol, Debjit Paul, Leo Laugier, Karl Aberer, Antoine Bosselut

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comparing Machines and Children: Using Developmental Psychology Experiments to Assess the Strengths and Weaknesses of LaMDA Responses

Eliza Kosoy, Emily Rose Reagan, Leslie Lai, Alison Gopnik, Danielle Krettek Cobb

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, 7 figures

Decoding the Enigma: Benchmarking Humans and AIs on the Many Facets of Working Memory

Ankur Sikarwar, Mengmi Zhang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments Conference on Neural Information Processing Systems (NeurIPS 2023)

Population-based Evaluation in Repeated Rock-Paper-Scissors as a Benchmark for Multiagent Reinforcement Learning

Marc Lanctot, John Schultz, Neil Burch, Max Olan Smith, Daniel Hennes, Thomas Anthony, Julien Perolat

专题命中 推理评测 ：planning(abstract)；分类 cs.AI、cs.LG

Comments 25 pages, 8 figures, Accepted at TMLR October 2023

Perception Test: A Diagnostic Benchmark for Multimodal Video Models

Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, João Carreira

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments 37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks

Improvement of a Prediction Model for Heart Failure Survival through Explainable Artificial Intelligence

Pedro A. Moreno-Sanchez

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments arXiv admin note: text overlap with arXiv:2105.10368

DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding

Xiao-Yu Guo, Yuan-Fang Li, Gholamreza Haffari

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 12 pages, 5 figures, EMNLP 2023 Long Paper

Are All Steps Equally Important? Benchmarking Essentiality Detection of Events

Haoyu Wang, Hongming Zhang, Yueguan Wang, Yuqian Deng, Muhao Chen, Dan Roth

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2023

Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting

Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, Jilin Chen

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments To appear at EMNLP 2023 main conference

CLAIR: Evaluating Image Captions with Large Language Models

David Chan, Suzanne Petryk, Joseph E. Gonzalez, Trevor Darrell, John Canny

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments To Appear at EMNLP 2023

API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2023

CorefPrompt: Prompt-based Event Coreference Resolution by Measuring Event Type and Argument Compatibilities

Sheng Xu, Peifeng Li, Qiaoming Zhu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP2023

Explaining Interactions Between Text Spans

Sagnik Ray Choudhury, Pepa Atanasova, Isabelle Augenstein

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments code: https://github.com/copenlu/spanex , dataset: https://huggingface.co/datasets/copenlu/spanex. Accepted EMNLP 2023

HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science

Yu Song, Santiago Miret, Huan Zhang, Bang Liu

专题命中 推理评测 ：verifier(abstract)；分类 cs.CL、cs.AI

EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation

Wang You, Wenshan Wu, Yaobo Liang, Shaoguang Mao, Chenfei Wu, Maosong Cao, Yuzhe Cai, Yiduo Guo, Yan Xia, Furu Wei, Nan Duan

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Yue Jiang, Eldon Schoop, Amanda Swearngin, Jeffrey Nichols

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark

Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, Andrey Evlampiev

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments to appear in EMNLP 2020

ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learning

Hosein Hasanbeig, Hiteshi Sharma, Leo Betthauser, Felipe Vieira Frujeri, Ida Momennejad

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments

Zhivar Sourati, Vishnu Priya Prasanna Venkatesh, Darshan Deshpande, Himanshu Rawlani, Filip Ilievski, Hông-Ân Sandlin, Alain Mermoud

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for Cloze-style Reading Comprehension

Shima Foolad, Kourosh Kiani

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments submitted for neurocomputing journal

Journal ref Neurocomputing Volume 558, 14 November 2023, 126786

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 19 pages

Are Deep Neural Networks SMARTer than Second Graders?

Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Kevin A. Smith, Joshua B. Tenenbaum

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments Extended version of CVPR 2023 paper. For the SMART-101 dataset, see http://smartdataset.github.io/smart101

WellXplain: Wellness Concept Extraction and Classification in Reddit Posts for Mental Health Analysis

Muskan Garg

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Beyond Object Recognition: A New Benchmark towards Object Concept Learning

Yong-Lu Li, Yue Xu, Xinyu Xu, Xiaohan Mao, Yuan Yao, Siqi Liu, Cewu Lu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments ICCV 2023. Webpage: https://mvig-rhos.com/ocl

Through the Lens of Core Competency: Survey on Evaluation of Large Language Models

Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng, Weinan Zhang, Ting Liu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, Orhan Firat

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments 19 pages

Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation

Ambrose Robinson, William Thorne, Ben P. Wu, Abdullah Pandor, Munira Essat, Mark Stevenson, Xingyi Song

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICCV 2023. Website: whoops-benchmark.github.io