arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.20195cs.CL
收藏

Monocle:基于不确定性主动学习的长文本生成混合局部-全局上下文评估

Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learning

Tsinghua University(清华大学);Beijing Jiaotong University(北京交通大学);Beijing University of Posts and Telecommunications(北京邮电大学);Xiamen University(厦门大学);Beijing Language and Culture University(北京语言大学)

Xiaorong Wang, Ting Yang, Zhu Zhang, Shuo Wang, Zihan Zhou, Liner Yang, Zhiyuan Liu, Maosong Sun

AI总结:

针对长文本评估随长度增加性能下降的问题,提出分而治之的局部-全局评估框架,结合混合上下文学习与基于不确定性的主动学习,降低标注成本并提升评估效果。

摘要:

评估模型生成的长篇文本的质量具有挑战性,即使使用先进的LLM-as-a-Judge方法,由于随着输入长度的增加性能会下降,评估难度依然很大。为了解决这个问题,我们提出了一种分而治之的方法,将综合评估任务分解为一系列局部评分任务,随后进行最终的全局评估。该策略允许进行更细粒度且更易管理的评估,确保文本的每个片段在连贯性和质量方面都被单独评估,同时兼顾整个文本的整体结构和一致性。此外,我们引入了一种混合上下文学习方法,利用人工标注来提升局部和全局评估的性能。通过将人工生成的反馈直接纳入评估过程,该方法使模型能够更好地与人类判断保持一致。最后,我们开发了一种基于不确定性的主动学习算法,该算法能够高效地选择用于人工标注的数据样本,从而在实际场景中降低标注成本。实验结果表明,所提出的评估框架优于几个代表性基线,凸显了我们方法的有效性。

英文摘要:

Assessing the quality of long-form, model-generated text is challenging, even with advanced LLM-as-a-Judge methods, due to performance degradation as input length increases. To address this issue, we propose a divide-and-conquer approach, which breaks down the comprehensive evaluation task into a series of localized scoring tasks, followed by a final global assessment. This strategy allows for more granular and manageable evaluations, ensuring that each segment of the text is assessed in isolation for both coherence and quality, while also accounting for the overall structure and consistency of the entire piece. Moreover, we introduce a hybrid in-context learning approach that leverages human annotations to enhance the performance of both local and global evaluations. By incorporating human-generated feedback directly into the evaluation process, this method allows the model to better align with human judgment. Finally, we develop an uncertainty-based active learning algorithm that efficiently selects data samples for human annotation, thereby reducing annotation costs in practical scenarios. Experimental results show that the proposed evaluation framework outperforms several representative baselines, highlighting the effectiveness of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑