arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.04210cs.AIcs.LG
收藏

通过可扩展的交互监督引导大语言模型

Steering LLMs via Scalable Interactive Oversight

Fudan University(复旦大学);Shanghai Qiji Zhifeng Co., Ltd.(上海齐济智峰有限公司);Shanghai Innovation Institute(上海创新研究院)

Enyu Zhou, Zhiheng Xi, Long Ma, Zhihao Zhang, Shihan Dou, Zhikai Lei, Guoteng Wang, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang

AI总结:

本文提出可扩展的交互监督框架,通过递归决策树提升人类对AI任务的引导能力,实现非专家生成高质量产品需求文档,并通过强化学习优化

摘要:

随着大型语言模型越来越多地自动化复杂、长周期的任务,如'氛围编码',出现了一个监督缺口。虽然模型在执行方面表现出色,但用户往往难以有效引导它们,原因包括领域专业知识不足、精确意图表达困难以及无法可靠验证复杂输出。这在可扩展的监督中提出了关键挑战:使人类能够负责任地引导超越其自身指定或验证能力的任务。为此,我们提出了可扩展的交互监督框架,将复杂意图分解为可管理决策的递归树,以增强人类监督。而不是依赖开放式的提示,我们的系统在每个节点提取低负担反馈,并递归地将这些信号聚合为精确的全局指导。在网页开发任务中验证了我们的框架,使非专家能够生成专家级的产品需求文档,实现了54%的对齐改进。关键的是,我们证明该框架可以通过强化学习仅使用在线用户反馈进行优化,为AI扩展时保持人类控制提供了实用路径。

英文摘要:

As Large Language Models increasingly automate complex, long-horizon tasks such as \emph{vibe coding}, a supervision gap has emerged. While models excel at execution, users often struggle to guide them effectively due to insufficient domain expertise, the difficulty of articulating precise intent, and the inability to reliably validate complex outputs. It presents a critical challenge in scalable oversight: enabling humans to responsibly steer AI systems on tasks that surpass their own ability to specify or verify. To tackle this, we propose Scalable Interactive Oversight, a framework that decomposes complex intent into a recursive tree of manageable decisions to amplify human supervision. Rather than relying on open-ended prompting, our system elicits low-burden feedback at each node and recursively aggregates these signals into precise global guidance. Validated in web development task, our framework enables non-experts to produce expert-level Product Requirement Documents, achieving a 54\% improvement in alignment. Crucially, we demonstrate that this framework can be optimized via Reinforcement Learning using only online user feedback, offering a practical pathway for maintaining human control as AI scales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑