大模型推理能力

Multimodal Foundation Models Exploit Text to Make Medical Image Predictions

Thomas Buckley, James A. Diao, Pranav Rajpurkar, Adam Rodman, Arjun K. Manrai

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark

Rong-Cheng Tu, Zi-Ao Ma, Tian Lan, Yuehao Zhao, Heyan Huang, Xian-Ling Mao

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modeling

Daehoon Gwak, Junwoo Park, Minho Park, Chaehun Park, Hyunchan Lee, Edward Choi, Jaegul Choo

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 Findings

Large Language Models on Graphs: A Comprehensive Survey

Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, Jiawei Han

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments 25 pages

Journal ref Transactions on Knowledge and Data Engineering (TKDE) 2024

GPT-4V Cannot Generate Radiology Reports Yet

Yuyang Jiang, Chacha Chen, Dang Nguyen, Benjamin M. Mervak, Chenhao Tan

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 24 pages, 3 figures, code: https://github.com/ChicagoHAI/cxr-eval-gpt-4v Findings paper presented at Machine Learning for Health (ML4H) symposium 2024, December 15-16, 2024, Vancouver, Canada, 26 pages

MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs

Mengyuan Zhang, Ruihui Wang, Bo Xia, Yuan Sun, Xiaobing Zhao

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering

Nghia Trung Ngo, Chien Van Nguyen, Franck Dernoncourt, Thien Huu Nguyen

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

LeKUBE: A Legal Knowledge Update BEnchmark

Changyue Wang, Weihang Su, Hu Yiran, Qingyao Ai, Yueyue Wu, Cheng Luo, Yiqun Liu, Min Zhang, Shaoping Ma

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

LLMs as Method Actors: A Model for Prompt Engineering and Architecture

Colin Doyle

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

RoCar: A Relationship Network-based Evaluation Method for Large Language Models

Ming Wang, Wenfang Wu, Chongyun Gao, Daling Wang, Shi Feng, Yifei Zhang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Abstract2Appendix: Academic Reviews Enhance LLM Long-Context Capabilities

Shengzhi Li, Kittipat Kampa, Rongyu Lin, Bohang Li, Shichao Pei

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments We share our latest dataset on https://github.com/findalexli/Abstract2Appendix

HourVideo: 1-Hour Video-Language Understanding

Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, Li Fei-Fei

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments NeurIPS 2024 Datasets and Benchmarks Track; 28 pages

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models

Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

What Really is Commonsense Knowledge?

Quyet V. Do, Junze Li, Tung-Duong Vuong, Zhaowei Wang, Yangqiu Song, Xiaojuan Ma

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Code and data will be released together with the next version of the paper

Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance

Niklas Wretblad, Oskar Holmström, Erik Larsson, Axel Wiksäter, Oscar Söderlund, Hjalmar Öhman, Ture Pontén, Martin Forsberg, Martin Sörme, Fredrik Heintz

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Explainable cognitive decline detection in free dialogues with a Machine Learning approach based on pre-trained Large Language Models

Francisco de Arriba-Pérez, Silvia García-Méndez, Javier Otero-Mosquera, Francisco J. González-Castaño

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Journal ref Applied Intelligence, 1-16 (2024)

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, Chenghua Lin, Wenhao Huang, Jie Fu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

MolCap-Arena: A Comprehensive Captioning Benchmark on Language-Enhanced Molecular Property Prediction

Carl Edwards, Ziqing Lu, Ehsan Hajiramezanali, Tommaso Biancalani, Heng Ji, Gabriele Scalia

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models

Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, Haodong Wang, Zhengyang Wang, Wenju Xu, Jingfeng Yang, Qingyu Yin, Xian Li, Priyanka Nigam, Yi Xu, Kai Chen, Qiang Yang, Meng Jiang, Bing Yin

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments NeurIPS 2024 Datasets and Benchmarks Track Accepted. Modified typos in Figure 9

Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities

Md Tauseef Alam, Raju Halder, Abyayananda Maiti

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.AI、cs.LG

KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge

Pengcheng Jiang, Lang Cao, Cao Xiao, Parminder Bhatia, Jimeng Sun, Jiawei Han

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2024

One Thousand and One Pairs: A "novel" challenge for long-context language models

Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, Mohit Iyyer

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024, camera ready

Evaluating Consistencies in LLM responses through a Semantic Clustering of Question Answering

Yanggyu Lee, Jihie Kim

专题命中 推理评测 ：CoT(abstract)；分类 cs.CL、cs.AI

Comments Accepted to the Trustworthy AI Workshop at IJCAI 2024

Causality for Large Language Models

Anpeng Wu, Kun Kuang, Minqin Zhu, Yingrong Wang, Yujia Zheng, Kairong Han, Baohong Li, Guangyi Chen, Fei Wu, Kun Zhang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential

Qing Lyu, Josh Tan, Michael E. Zapadka, Janardhana Ponnatapura, Chuang Niu, Kyle J. Myers, Ge Wang, Christopher T. Whitlow

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework

Fan Liu, Yue Feng, Zhao Xu, Lixin Su, Xinyu Ma, Dawei Yin, Hao Liu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Graph Neural Network Enhanced Retrieval for Question Answering of LLMs

Zijian Li, Qingyan Guo, Jiawei Shao, Lei Song, Jiang Bian, Jun Zhang, Rui Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Under review

Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models

Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, Qi Liu

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.LG

Anchored Alignment for Self-Explanations Enhancement

Luis Felipe Villa-Arenas, Ata Nizamoglu, Qianli Wang, Sebastian Möller, Vera Schmitt

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Understanding the Role of LLMs in Multimodal Evaluation Benchmarks

Botian Jiang, Lei Li, Xiaonan Li, Zhaowei Li, Xiachong Feng, Lingpeng Kong, Qi Liu, Xipeng Qiu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI