发表机构
NTT, Inc.（NTT公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文针对对话生成产物的修订传播问题，引入新基准并评估9种方法，发现从3个并行样本中选择的高性价比方法可提升准确率2.2%至9.7%。
AI 中文摘要
大型语言模型（LLMs）常通过对话中的迭代生成与修订循环帮助用户生成产物。此处的挑战在于，当用户在修订时仅指定局部改动，LLMs需识别相关依赖并将修订传播至产物的所有受影响部分。本文研究LLMs在对话生成产物上的此能力，这类产物的上下文及其依赖可能嵌入在对话历史中。为实现实际应用，我们还探索该新场景下的高性价比测试时计算。具体而言，我们为此场景引入新基准，使用gpt-oss-20b/120b、gpt-5.4-mini及qwen3.5-9b/27b/122b在该基准上评估9种修订方法，包括顺序反思与并行采样变体。结果显示，基线方法的准确率为68.3%至93%，最具性价比的方法是通过基于LLM的选择或medoid选择从3个并行样本中挑选，其准确率提升2.2%至9.7%。我们的代码与数据集可在此httpsURL获取。
英文摘要
Large Language Models (LLMs) often help users generate artifacts through iterative cycles of generation and revision in conversation. A challenge here is that, when users specify only a local change during revision, LLMs must instead identify the relevant dependencies and propagate the revision to all affected parts of the artifact. This paper studies this ability of LLMs on conversationally generated artifacts, where the artifact context and its dependencies may be embedded in the conversation history. Toward practical use, we also explore cost-effective test-time compute for this new setting. Specifically, we introduce a new benchmark for this setting, and evaluate nine revision methods, including sequential reflection and parallel sampling variants, using gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b on the benchmark. The results show that baselines achieve accuracies of 68.3--93%, and the most cost-effective method is selecting from three parallel samples using either LLM-based or medoid selection, which improves accuracy by 2.2--9.7%. Our code and dataset are available at https://github.com/ntt-dkiku/llm-revision-propagation.
CommentsAccepted at EMNLP 2026 Industry Track. The code is available at https://github.com/ntt-dkiku/llm-revision-propagation
