发表机构
University of Canterbury; Unitec Institute of Technology; AgResearch Ltd.（坎特伯雷大学; Unitec理工学院; 新西兰农业研究有限公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究评估提示扰动对大型语言模型在决策任务中偏见与幻觉的影响，发现扰动可缓解部分模型的问题，Claude 3表现优于GPT3.5，强调严格测试对可靠AI助手的重要性。
AI 中文摘要
大型语言模型（LLMs）在各种自然语言处理任务中展现出了卓越的能力，导致其作为智能助手在决策情境中被广泛部署。然而，这些模型日益增长的复杂性引发了对其可靠性的担忧，特别是在偏见和幻觉方面。在本研究中，我们评估了LLMs对决策任务中原始询问的扰动变体的鲁棒性。我们发现，与以往研究相反，扰动可以在某些LLMs中缓解偏见和幻觉，而在其他模型中则不然。研究发现，Claude 3在大多数数据集所代表的任务中更为有效，而像GPT3.5这样的模型则表现出不同程度的适用性，在某些情况下表现相当，但在其他情况下则显著落后。这些见解对于理解在现实应用中部署基于LLM的助手作为有效决策支持工具的实际意义至关重要，强调了进行严格测试和验证以确保可靠性和有效性的必要性。本研究为不断增长的LLM评估研究体系做出了贡献，并为在关键决策情境中开发更稳健和可信的AI助手提供了见解。
英文摘要
Large language models (LLMs) have shown remarkable capabilities in various natural language processing tasks, leading to their widespread deployment as intelligent assistants in decision-making contexts. However, the increasing complexity of these models raises concerns about their reliability, particularly regarding bias and hallucination. In this work, we evaluate the robustness of LLMs to perturbed variations of the original inquiry in decision-making tasks. We show that contrary to previous studies, perturbations can mitigate bias and hallucination in some LLMs over other models. It's found that Claude 3 is more effective for the tasks represented in most datasets, whereas models like GPT3.5 exhibit varying levels of adequacy, performing comparably in some cases but falling significantly behind in others. These insights are crucial for understanding the practical implications of deploying LLM-based assistants as effective decision-support tools in real-world applications, emphasising the need for rigorous testing and validation to ensure reliability and effectiveness. This study contributes to the growing body of research on LLM evaluation and provides insights for developing more robust and trustworthy AI assistants in critical decision-making contexts.
Comments14 pages. Published in ICONIP 2024 (Neural Information Processing), LNCS 15290, Springer Nature, 2025
Journal refNeural Information Processing (ICONIP 2024), Lecture Notes in Computer Science (LNCS), vol. 15290, pp. 361-374, Springer, 2025
DOI：10.1007/978-981-96-6588-4_25
