通过离线模拟与LLM实现软件脚本自动化中的技能发现
Skill Discovery for Software Scripting Automation via Offline Simulations with LLMs
AI总结：
本文提出了一种利用离线模拟与LLM构建软件特定技能集的方法，通过生成和验证脚本提升自动化效率，减少运行时成本。
摘要：
脚本接口使用户能够自动化任务并自定义软件工作流，但创建脚本传统上需要编程专业知识和对特定API的熟悉，这对许多用户来说是一个障碍。尽管大型语言模型（LLM）可以从自然语言查询生成代码，但运行时代码生成受到未验证代码、安全风险、响应时间更长和计算成本更高等因素的严重限制。为弥合这一差距，我们提出了一种离线模拟框架，通过利用LLM和公开可用的脚本指南，来构建一个软件特定的技能集，即一组经过验证的脚本。我们的框架包括两个组成部分：（1）任务创建，利用自上而下的功能指导和自下而上的API协同探索来生成有用的任务；（2）技能生成与试验，通过执行反馈来精炼和验证脚本。为了高效地导航广泛的API景观，我们引入了基于图神经网络（GNN）的链接预测模型，以捕捉API协同效应，从而生成涉及未充分利用的API的技能，并扩大技能集的多样性。在Adobe Illustrator上的实验表明，我们的框架显著提高了自动化成功率，减少了响应时间，并节省了运行时令牌成本，与传统运行时代码生成相比。这是首次尝试将软件脚本接口作为LLM系统测试平台，突显了在受控环境中利用执行反馈的优势，并提供了有价值的见解，以在专用软件领域内对齐AI能力与用户需求。
英文摘要：
Scripting interfaces enable users to automate tasks and customize software workflows, but creating scripts traditionally requires programming expertise and familiarity with specific APIs, posing barriers for many users. While Large Language Models (LLMs) can generate code from natural language queries, runtime code generation is severely limited due to unverified code, security risks, longer response times, and higher computational costs. To bridge the gap, we propose an offline simulation framework to curate a software-specific skillset, a collection of verified scripts, by exploiting LLMs and publicly available scripting guides. Our framework comprises two components: (1) task creation, using top-down functionality guidance and bottom-up API synergy exploration to generate helpful tasks; and (2) skill generation with trials, refining and validating scripts based on execution feedback. To efficiently navigate the extensive API landscape, we introduce a Graph Neural Network (GNN)-based link prediction model to capture API synergy, enabling the generation of skills involving underutilized APIs and expanding the skillset's diversity. Experiments with Adobe Illustrator demonstrate that our framework significantly improves automation success rates, reduces response time, and saves runtime token costs compared to traditional runtime code generation. This is the first attempt to use software scripting interfaces as a testbed for LLM-based systems, highlighting the advantages of leveraging execution feedback in a controlled environment and offering valuable insights into aligning AI capabilities with user needs in specialized software domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
