arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01384cs.CL
收藏

可读性水平控制下的自由文本理由生成

Free-text Rationale Generation under Readability Level Control

German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Computational Linguistics, Department of Linguistics, Universität Potsdam(波茨坦大学语言学系计算语言学部门);Quality and Usability Lab, Technische Universität Berlin(柏林技术大学质量与可用性实验室);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究院);Computer Science Institute, Hochschule Ruhr West(鲁尔西部大学计算机科学研究所)

Yi-Sheng Hsu, Nils Feldhus, Sherzod Hakimov

AI总结:

本文研究大语言模型在可读性水平控制下的自由文本理由生成,发现解释可适应指令但复杂度不完全匹配传统指标,且中等复杂度理由质量较高,高中水平可读性最受青睐。

摘要:

自由文本理由以自然语言为模型决策提供依据,因此在众多任务的解释方法中变得易于理解且广受欢迎。然而,其有效性可能因误读和幻觉而受到削弱。作为一项扰动测试,我们研究了大语言模型(LLMs)在可读性水平控制影响下执行理由生成的表现,即要求模型针对特定专业水平(如六年级或大学水平)生成解释。我们发现,解释能够适应此类指令,尽管所观察到的可读性水平之间的差异并未完全匹配传统可读性指标所定义的复杂度分数。此外,生成的理由往往呈现中等复杂度,这与使用自动指标测量的质量相关。最后,我们的人工标注者证实,在所有可读性水平上,理由总体上给人满意的印象,其中高中水平的可读性最常被感知且最受青睐。

英文摘要:

Free-text rationales justify model decisions in natural language and thus become likable and accessible among approaches to explanation across many tasks. However, their effectiveness can be hindered by misinterpretation and hallucination. As a perturbation test, we investigate how large language models (LLMs) perform rationale generation under the effects of readability level control, i.e., being prompted for an explanation targeting a specific expertise level, such as sixth grade or college. We find that explanations are adaptable to such instruction, though the observed distinction between readability levels does not fully match the defined complexity scores according to traditional readability metrics. Furthermore, the generated rationales tend to feature medium level complexity, which correlates with the measured quality using automatic metrics. Finally, our human annotators confirm a generally satisfactory impression on rationales at all readability levels, with high-school-level readability being most commonly perceived and favored.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑