当今LLM研究中的亮点、法则与缺陷
Awes, Laws, and Flaws From Today's LLM Research
AI总结:
本文批判性审查了当代LLM研究的科学方法论,通过评估2020至2024年间2000多项研究,发现伦理声明减少、LLM作为评估者兴起等趋势,并提出改进建议。
摘要:
我们对当代大型语言模型(LLM)研究的科学方法论进行了批判性审查。为此,我们基于通常被认为是优秀研究的标准(例如统计检验的存在和可重复性),评估了2020年至2024年间发布的2000多项研究工作,并用处于争议中心的论点(例如涌现行为的声明)对其进行了交叉验证。我们发现了多种趋势,例如伦理免责声明的减少、LLM作为评估者的兴起,以及在不利用人类评估的情况下增加对LLM推理能力的声明。我们注意到,会议审查清单在遏制其中一些问题方面是有效的,但在研究中平衡速度和严谨性不能仅仅依赖这些。我们将所有这些发现与近期元综述的发现联系起来,并扩展了关于如何解决LLM研究中哪些有效、哪些无效以及哪些应该有效的建议。
英文摘要:
We perform a critical examination of the scientific methodology behind contemporary large language model (LLM) research. For this we assess over 2,000 research works released between 2020 and 2024 based on criteria typical of what is considered good research (e.g. presence of statistical tests and reproducibility), and cross-validate it with arguments that are at the centre of controversy (e.g., claims of emergent behaviour). We find multiple trends, such as declines in ethics disclaimers, a rise of LLMs as evaluators, and an increase on claims of LLM reasoning abilities without leveraging human evaluation. We note that conference checklists are effective at curtailing some of these issues, but balancing velocity and rigour in research cannot solely rely on these. We tie all these findings to findings from recent meta-reviews and extend recommendations on how to address what does, does not, and should work in LLM research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。