确定性与LLM控制的编译流程在COBOL到Python现代化中的对比
Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization
AI总结:
本文通过实验对比确定性和LLM控制的编译流程,发现确定性方法在准确性、鲁棒性和效率上更优,且成本更低。
摘要:
遗留COBOL系统现代化仍面临专家稀缺、代码库庞大且长期存在以及严格正确性要求的挑战。最近基于大语言模型(LLM)的现代化系统越来越多地依赖代理工作流,其中模型控制多步骤工具执行。然而,将执行控制委托给LLM是否能提高结构软件工程工作流的正确性、鲁棒性和效率仍不清楚。本文通过受控实验证明了确定性和LLM控制的编译流程。使用统一的实验框架,我们保持语言模型、提示、工具、配置和源程序不变,仅改变执行控制策略。这将编排作为唯一实验变量。我们通过功能正确性、跨重复随机运行的鲁棒性和计算效率评估两种方法。在多个模型中,确定性编排在计算准确性上与LLM控制编排相当,同时提高了最坏情况的鲁棒性并减少了运行间的性能变化。确定性执行还通过3.5倍减少token消耗,导致显著降低的操作成本。这些结果表明,在具有明确验证阶段的结构化现代化工作流中,固定执行策略比无减少翻译质量的完全代理编排更稳定且成本更低。
英文摘要:
Modernizing legacy COBOL systems remains difficult due to scarce expertise, large and long-lived codebases, and strict correctness requirements. Recent large language model (LLM)-based modernization systems increasingly rely on agentic workflows in which the model controls multi-step tool execution. However, it remains unclear whether delegating execution control to the LLM improves correctness, robustness, or efficiency in structured software engineering workflows. We present a controlled empirical study of deterministic and LLM-controlled orchestration for COBOL-to-Python modernization. Using a unified experimental framework, we hold the language models, prompts, tools, configurations, and source programs constant while varying only the execution control strategy. This isolates orchestration as the sole experimental variable. We evaluate both approaches using functional correctness, robustness across repeated stochastic runs, and computational efficiency. Across multiple models, deterministic orchestration achieves comparable computational accuracy to LLM-controlled orchestration while improving worst-case robustness and reducing performance variability across runs. Deterministic execution also reduces token consumption by up to 3.5x, leading to substantially lower operational cost. These results suggest that, in structured modernization workflows with explicit validation stages, fixed execution policies provide more stable and cost-efficient behavior than fully agentic orchestration without reducing translation quality.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。