arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.14037cs.CLcs.AI
收藏

LLM能取代Neil deGrasse Tyson吗?评估LLM作为科学传播者的可靠性

Can LLMs replace Neil deGrasse Tyson? Evaluating the Reliability of LLMs as Science Communicators

Indian Institute of Technology, Delhi(印度理工学院德里分校)

Prasoon Bajpai, Niladri Chatterjee, Subhabrata Dutta, Tanmoy Chakraborty

AI总结:

本文评估LLMs作为科学传播者的可靠性,引入包含742个是/否查询的SCiPS-QA数据集和基准套件,发现Llama-3-70B可媲美GPT-4 Turbo,并揭示GPT模型在验证响应及人类评估易受欺骗的问题。

摘要:

大型语言模型(LLMs)及由其驱动的AI助手在专家和业余用户中的使用量正呈指数级增长。在这项工作中,我们专注于评估当前LLMs作为科学传播者的可靠性。与现有基准不同,我们的方法强调在需要对可答性有细致理解和意识的科学问答任务上评估这些模型。我们引入了一个新颖的数据集SCiPS-QA,包含742个嵌入复杂科学概念中的是/否查询,以及一个在各种标准下评估LLMs正确性和一致性的基准套件。我们对来自OpenAI GPT家族的三个专有LLMs和来自Meta Llama-2、Llama-3及Mistral家族的13个开源LLMs进行了基准测试。尽管大多数开源模型的表现显著逊于GPT-4 Turbo,但我们的实验确定Llama-3-70B是一个强有力的竞争者,在各个评估方面经常超越GPT-4 Turbo。我们还发现,即使是GPT模型在可靠验证LLM响应方面也表现出普遍的无能。此外,我们观察到一个令人担忧的趋势,即人类评估者会被GPT-4 Turbo的错误回答所欺骗。

英文摘要:

Large Language Models (LLMs) and AI assistants driven by these models are experiencing exponential growth in usage among both expert and amateur users. In this work, we focus on evaluating the reliability of current LLMs as science communicators. Unlike existing benchmarks, our approach emphasizes assessing these models on scientific questionanswering tasks that require a nuanced understanding and awareness of answerability. We introduce a novel dataset, SCiPS-QA, comprising 742 Yes/No queries embedded in complex scientific concepts, along with a benchmarking suite that evaluates LLMs for correctness and consistency across various criteria. We benchmark three proprietary LLMs from the OpenAI GPT family and 13 open-access LLMs from the Meta Llama-2, Llama-3, and Mistral families. While most open-access models significantly underperform compared to GPT-4 Turbo, our experiments identify Llama-3-70B as a strong competitor, often surpassing GPT-4 Turbo in various evaluation aspects. We also find that even the GPT models exhibit a general incompetence in reliably verifying LLM responses. Moreover, we observe an alarming trend where human evaluators are deceived by incorrect responses from GPT-4 Turbo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑