arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03619cs.CL
收藏

从人类偏好中学习查询特定评分标准用于DeepResearch报告生成

Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation

Tencent(腾讯);Fudan University(复旦大学);Tsinghua University(清华大学)
Changze Lv, Jie Zhou, Wentao Zhao, Jingwen Xu, Shihan Dou, Zisu Huang, Muzhao Tian, Xiaohua Wang, Zhengkang Guo, Yang Liu, Pluto Zhou, Tao Gui, Le Tian, Xiao Zh… 展开作者

Changze Lv, Jie Zhou, Wentao Zhao, Jingwen Xu, Shihan Dou, Zisu Huang, Muzhao Tian, Xiaohua Wang, Zhengkang Guo, Yang Liu, Pluto Zhou, Tao Gui, Le Tian, Xiao Zhou, Xiaoqing Zheng, Xuanjing Huang, Jie Zhou

AI总结:

提出一种通过强化学习训练查询特定评分标准生成器的流水线,以解决DeepResearch长报告生成中缺乏可验证奖励信号的问题,并在人类偏好测试和下游任务中取得显著性能提升。

摘要:

如今,开发可靠的DeepResearch式长文本报告生成仍然具有挑战性,因为训练和评估缺乏可验证的奖励信号。因此,基于评分标准的评估已成为常见做法。然而,现有方法要么依赖于粗粒度、预定义的评分标准,缺乏足够的细粒度,要么依赖于人工构建的查询特定评分标准,成本高昂且难以扩展。在本文中,我们提出了一种流水线,用于训练偏好基础的查询特定评分标准生成器,专门用于DeepResearch报告生成。我们首先构建了一个DeepResearch式查询数据集,其中包含成对报告上的人类偏好注释,并通过强化学习训练评分标准生成器,使用结合偏好一致性、格式有效性和基于LLM的评分标准评估的混合奖励。我们在两个阶段评估生成的评分标准生成器。首先,在保留的人类偏好测试集上,学习到的评分标准比通用的、提示的或SFT训练的评分标准更有效地区分偏好报告和拒绝报告。其次,当用作训练DeepResearch系统的奖励信号时,我们的评分标准生成器在简单的单智能体ReAct框架和复杂的多智能体工作流上,在DeepResearch Bench上均取得了显著的性能提升。

英文摘要:

Nowadays, developing reliable DeepResearch-style long-form report generation remains challenging, as training and evaluation lack verifiable reward signals. Accordingly, rubric-based evaluation has become a common practice. However, existing approaches either rely on coarse, pre-defined rubrics that lack sufficient granularity or depend on manually constructed query-specific rubrics that are costly and difficult to scale. In this paper, we propose a pipeline to train preference-grounded query-specific rubric generators tailored for DeepResearch report generation. We first construct a dataset of DeepResearch-style queries annotated with human preferences over paired reports, and train rubric generators via reinforcement learning with a hybrid reward combining preference consistency, format validity, and LLM-based rubric evaluation. We evaluate the resulting rubric generators in two stages. First, on a held-out human-preference test set, the learned rubrics discriminate preferred from rejected reports more effectively than generic, prompted, or SFT-trained rubric alternatives. Second, when used as reward signals to train DeepResearch systems, our rubric generators yield substantial performance gains under both a simple single-agent ReAct framework and a complex multi-agent workflow on the DeepResearch Bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑