扩展不可验证的奖励：视觉洞察的案例研究
Scaling Unverifiable Rewards: A Case Study on Visual Insights
AI总结：
本文提出Selective TTS框架，通过多阶段流程优化提升视觉洞察质量，实现计算预算固定下的性能提升。
摘要：
大型语言模型（LLM）代理可以通过测试时间扩展（TTS）逐步自动化复杂推理，这种推理通过奖励信号进行迭代优化。然而，许多现实任务涉及多阶段流程，其最终结果缺乏可验证的奖励或足够的数据来训练稳健的奖励模型，使基于判断的优化容易在阶段间积累误差。我们提出了一种基于过程的优化框架Selective TTS，该框架在多代理流程中跨不同阶段扩展推理，而不是像先前工作那样通过时间重复优化。通过在阶段间分配计算并使用特定于过程的判断者修剪低质量分支，Selective TTS减轻了判断漂移并稳定了优化。基于数据科学流程，我们构建了一个端到端的多代理流程，用于生成给定数据集的视觉洞察图表和报告，并设计了一个可靠的基于LLM的判断模型，与人类专家一致（Kendall's τ=0.55）。我们提出的Selective TTS在固定计算预算下提高了洞察质量，平均分数从61.64增加到65.86，同时减少了方差。我们希望我们的发现成为扩展具有不可验证奖励的复杂、开放性任务的第一步，例如科学发现和故事生成。
英文摘要：
Large Language Model (LLM) agents can increasingly automate complex reasoning through Test-Time Scaling (TTS), iterative refinement guided by reward signals. However, many real-world tasks involve multi-stage pipeline whose final outcomes lack verifiable rewards or sufficient data to train robust reward models, making judge-based refinement prone to accumulate error over stages. We propose Selective TTS, a process-based refinement framework that scales inference across different stages in multi-agent pipeline, instead of repeated refinement over time by prior work. By distributing compute across stages and pruning low-quality branches early using process-specific judges, Selective TTS mitigates the judge drift and stabilizes refinement. Grounded in the data science pipeline, we build an end-to-end multi-agent pipeline for generating visually insightful charts and report of given dataset, and design a reliable LLM-based judge model, aligned with human experts (Kendall's τ=0.55). Our proposed selective TTS then improves insight quality under a fixed compute budget, increasing mean scores from 61.64 to 65.86 while reducing variance. We hope our findings serve as the first step toward to scaling complex, open-ended tasks with unverifiable rewards, such as scientific discovery and story generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
