ChatGPT在推理、幻觉与交互性方面的多任务、多语言、多模态评估
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
AI总结：
本文基于23个数据集和自建多模态数据集，系统评估ChatGPT的多任务、多语言、多模态能力，发现其在多数任务上优于零样本大语言模型，但推理可靠性不足且存在幻觉问题，同时验证了多轮提示工程可借助人类协作提升模型性能。
摘要：
本文提出了一个利用公开可用数据集对ChatGPT等交互式大语言模型（LLMs）进行定量评估的框架。我们使用涵盖8种不同常见自然语言处理（NLP）应用任务的23个数据集，对ChatGPT进行了广泛的技术评估。基于这些数据集以及一个新设计的多模态数据集，我们评估了ChatGPT的多任务、多语言和多模态能力。我们发现，ChatGPT在大多数任务上优于采用零样本学习的大语言模型，甚至在某些任务上优于经过微调的模型。我们发现它在理解非拉丁字母语言方面比生成这些语言表现得更好。它能够通过中间代码生成步骤，从文本提示生成多模态内容。此外，我们发现ChatGPT在逻辑推理、非文本推理和常识推理下的10个不同推理类别中平均准确率为63.41%，因此它是一个不可靠的推理器。例如，它更擅长演绎推理而非归纳推理。ChatGPT与其他大语言模型一样存在幻觉问题，并且由于无法访问外部知识库，它会从其参数化记忆中生成更多的外在幻觉。最后，ChatGPT的交互特性使得人类能够与底层大语言模型协作，以多轮“提示工程”的方式提高其性能，即在摘要任务上提升8%的ROUGE-1分数，在机器翻译任务上提升2%的ChrF++分数。我们还发布了用于评估集提取的代码库。
英文摘要：
This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn "prompt engineering" fashion. We also release codebase for evaluation set extraction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
