arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.07355cs.CL
收藏

共同思考,更好协作:结合人类与LLM的出声思考结果以实现有效的文本评估

Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation

Graduate School of Data Science, KAIST(韩国科学技术院数据科学研究生院);Department of Industrial & Systems Engineering, KAIST(韩国科学技术院工业与系统工程系)

SeongYeub Chu, JongWoo Kim, MunYong Yi

AI总结:

提出InteractEval框架,通过出声思考方法融合人类与LLM生成文本评估检查表属性,在四个维度上优于基线,并揭示二者在内部质量与外部对齐属性识别上的互补优势。

摘要:

本研究提出了InteractEval,这是一个利用出声思考(Think-Aloud,TA)方法整合人类专业知识与大语言模型(LLM)的框架,用于为基于检查表的文本评估生成属性。通过将人类的灵活性和推理能力与LLM的一致性相结合,InteractEval在连贯性、流畅性、一致性和相关性四个不同维度上优于传统的非基于LLM和基于LLM的基线。实验还考察了TA方法的有效性,表明它能促进人类和LLM的发散性思维,从而生成范围更广的相关属性并提升文本评估性能。比较分析显示,人类擅长识别与内部质量(连贯性和流畅性)相关的属性,而LLM在与外部对齐(一致性和相关性)相关的属性上表现更好。因此,同时利用人类和LLM可产生最佳评估结果。换言之,本研究强调在自动化的基于检查表的文本评估框架中有效结合人类与LLM的必要性。代码可在https://github.com/BBeeChu/InteractEval.git获取。

英文摘要:

This study introduces \textbf{InteractEval}, a framework that integrates human expertise and Large Language Models (LLMs) using the Think-Aloud (TA) method to generate attributes for checklist-based text evaluation. By combining human flexibility and reasoning with LLM consistency, InteractEval outperforms traditional non-LLM-based and LLM-based baselines across four distinct dimensions, consisting of Coherence, Fluency, Consistency, and Relevance. The experiment also investigates the effectiveness of the TA method, showing that it promotes divergent thinking in both humans and LLMs, leading to the generation of a wider range of relevant attributes and enhance text evaluation performance. Comparative analysis reveals that humans excel at identifying attributes related to internal quality (Coherence and Fluency), but LLMs perform better at those attributes related to external alignment (Consistency and Relevance). Consequently, leveraging both humans and LLMs together produces the best evaluation outcomes. In other words, this study emphasizes the necessity of effectively combining humans and LLMs in an automated checklist-based text evaluation framework. The code is available at \textbf{\url{https://github.com/BBeeChu/InteractEval.git}}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑