大模型推理能力

Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?

Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, Mohammad Hossein Rohban

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

ChatASU: Evoking LLM's Reflexion to Truly Understand Aspect Sentiment in Dialogues

Yiding Liu, Jingjing Wang, Jiamin Luo, Tao Zeng, Guodong Zhou

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents

Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, Carolin Lawrence

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted at the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2024)

Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level

Chenxu Wang, Bin Dai, Huaping Liu, Baoyuan Wang

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials

Mael Jullien, Marco Valentino, André Freitas

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Ever-Evolving Memory by Blending and Refining the Past

Seo Hyun Kim, Keummin Ka, Yohan Jo, Seung-won Hwang, Dongha Lee, Jinyoung Yeo

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 17 pages, 4 figures, 7 tables

Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra

Darioush Kevian, Usman Syed, Xingang Guo, Aaron Havens, Geir Dullerud, Peter Seiler, Lianhui Qin, Bin Hu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs

Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, James M. Rehg, Ziran Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments CVPR 2024

Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT

Amirhossein Abaskohi, Sara Baruni, Mostafa Masoudi, Nesa Abbasi, Mohammad Hadi Babalou, Ali Edalat, Sepehr Kamahi, Samin Mahdizadeh Sani, Nikoo Naghavian, Danial Namazifard, Pouya Sadeghi, Yadollah Yaghoobzadeh

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments 14 pages, 1 figure, 6 tables, Proceeding of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)

Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark

Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, David Jurgens

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Camera-ready version for EMNLP'23. First two authors contributed equally

DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries

Manit Mishra, Abderrahman Braham, Charles Marsom, Bryan Chung, Gavin Griffin, Dakshesh Sidnerlikar, Chatanya Sarin, Arjun Rajaram

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments 5 pages, Submitted to International Conference on AI in Cybersecurity

An Examination of the Compositionality of Large Generative Vision-Language Models

Teli Ma, Rong Li, Junwei Liang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

GlitchBench: Can large multimodal models detect video game glitches?

Mohammad Reza Taesiri, Tianjun Feng, Anh Nguyen, Cor-Paul Bezemer

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments CVPR 2024

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, Lu Hou

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments CVPR 2024 camera-ready version, code is available at https://github.com/RenShuhuai-Andy/TimeChat

Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, Jiaya Jia

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Code and models are available at https://github.com/dvlab-research/MiniGemini

Retrieval-Augmented Generation for Large Language Models: A Survey

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, Haofen Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Ongoing Work

From Graph to Word Bag: Introducing Domain Knowledge to Confusing Charge Prediction

Ang Li, Qiangchao Chen, Yiquan Wu, Ming Cai, Xiang Zhou, Fei Wu, Kun Kuang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

FunQA: Towards Surprising Video Comprehension

Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, Ziwei Liu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Project Page: https://funqa-benchmark.github.io/ Codebase: https://github.com/Jingkang50/FunQA

Polaris: A Safety-focused LLM Constellation Architecture for Healthcare

Subhabrata Mukherjee, Paul Gamble, Markel Sanz Ausin, Neel Kant, Kriti Aggarwal, Neha Manjunath, Debajyoti Datta, Zhengliang Liu, Jiayuan Ding, Sophia Busacca, Cezanne Bianco, Swapnil Sharma, Rae Lasko, Michelle Voisard, Sanchay Harneja, Darya Filippova, Gerry Meixiong, Kevin Cha, Amir Youssefi, Meyhaa Buvanesh, Howard Weingram, Sebastian Bierman-Lytle, Harpreet Singh Mangat, Kim Parikh, Saad Godil, Alex Miller

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Compressing LLMs: The Truth is Rarely Pure and Never Simple

Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, Yinfei Yang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ICLR 2024

Analyzing and Mitigating Object Hallucination in Large Vision-Language Models

Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, Huaxiu Yao

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments Accepted by ICLR 2024

Unmasking the Shadows of AI: Investigating Deceptive Capabilities in Large Language Models

Linge Guo

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments AI deception, Large Language Models, ChatGPT

Zero-shot and Few-shot Generation Strategies for Artificial Clinical Records

Erlend Frayling, Jake Lever, Graham McDonald

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.LG

Comments 4 pages

A Logical Pattern Memory Pre-trained Model for Entailment Tree Generation

Li Yuan, Yi Cai, Haopeng Ren, Jiexin Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted By Coling 2024

CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models

Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, Yifan Liu, Jingkuan Wang, Siyuan Qi, Kangning Zhang, Weinan Zhang, Yong Yu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 33pages

Acceleron: A Tool to Accelerate Research Ideation

Harshit Nigam, Manasi Patwardhan, Lovekesh Vig, Gautam Shroff

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted at AI2ASE Workshop at AAAI'24 Conference. 13 Pages and 4 Figures

Exploring LLM-based Agents for Root Cause Analysis

Devjeet Roy, Xuchao Zhang, Rashi Bhave, Chetan Bansal, Pedro Las-Casas, Rodrigo Fonseca, Saravan Rajmohan

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

ChatCite: LLM Agent with Human Workflow Guidance for Comparative Literature Summary

Yutong Li, Lu Chen, Aiwei Liu, Kai Yu, Lijie Wen

专题命中 推理评测 ：CoT(abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 5 figures

Pre-trained Language Model with Prompts for Temporal Knowledge Graph Completion

Wenjie Xu, Ben Liu, Miao Peng, Xu Jia, Min Peng

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted to Findings of ACL 2023

PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Code and Data released at https://github.com/pkunlp-icler/PCA-EVAL. Leaderboard at: https://docs.qq.com/sheet/DVUd4WUpGRHRqUnNV. This article supersedes its workshop version arxiv: 2310.02071. arXiv admin note: text overlap with arXiv:2310.02071