Ansible Lightspeed 代码补全服务使用情况的洞察
Insights from the Usage of the Ansible Lightspeed Code Completion Service
AI总结:
本文介绍了面向 Ansible 领域特定语言的 LLM 代码补全服务 Lightspeed,并基于上万名真实用户数据评估其效用,首次报告 N 日留存率并提出强接受率指标,证明小型专用模型在特定领域具有良好效果。
摘要:
能够生成代码的大语言模型(LLMs)的出现,使得创建提高开发者生产力的工具成为可能。开发者用于编写软件的集成开发环境(IDE)常被用作与 LLMs 交互的接口。尽管许多此类工具已经发布,但几乎都专注于通用编程语言。领域特定语言,例如对信息技术(IT)自动化至关重要的那些语言,尚未受到太多关注。Ansible 就是这样一种基于 YAML 的 IT 自动化特定语言。Ansible Lightspeed 是一项基于 LLM 的服务,专门设计用于根据自然语言提示生成 Ansible YAML。在本文中,我们介绍了 Ansible Lightspeed 服务的设计与实现。随后,我们使用多种指标评估其对开发者的效用,包括扩展使用情况、对用户编辑建议的分析以及用户情感分析。该评估基于收集自 10,696 名真实用户的数据,其中包括 3,910 名回访用户。Ansible Lightspeed 服务的代码和分析框架已公开供他人使用。据我们所知,我们的研究是首个涉及数千名领域特定语言代码助手用户的研究。我们也是首个展示 N 日用户留存率的代码补全工具,第 30 天的留存率为 13.66%。我们提出了用户接受率的改进版本,称为强接受率(Strong Acceptance rate),其中只有当建议被编辑的部分少于 50% 且这些编辑不改变建议的关键部分时,该建议才被视为被接受。通过专注于 Ansible,Lightspeed 能够为多行 Ansible 任务建议实现 49.08% 的强接受率。凭借我们的发现,我们为小型专用模型在领域特定环境中的有效性提供了见解。
英文摘要:
The availability of Large Language Models (LLMs) which can generate code, has made it possible to create tools that improve developer productivity. Integrated development environments or IDEs which developers use to write software are often used as an interface to interact with LLMs. Although many such tools have been released, almost all of them focus on general-purpose programming languages. Domain-specific languages, such as those crucial for Information Technology (IT) automation, have not received much attention. Ansible is one such YAML-based IT automation-specific language. Ansible Lightspeed is an LLM-based service designed explicitly to generate Ansible YAML, given natural language prompt. In this paper, we present the design and implementation of the Ansible Lightspeed service. We then evaluate its utility to developers using diverse indicators, including extended utilization, analysis of user edited suggestions, as well as user sentiments analysis. The evaluation is based on data collected for 10,696 real users including 3,910 returning users. The code for Ansible Lightspeed service and the analysis framework is made available for others to use. To our knowledge, our study is the first to involve thousands of users of code assistants for domain-specific languages. We are also the first code completion tool to present N-Day user retention figures, which is 13.66% on Day 30. We propose an improved version of user acceptance rate, called Strong Acceptance rate, where a suggestion is considered accepted only if less than 50% of it is edited and these edits do not change critical parts of the suggestion. By focusing on Ansible, Lightspeed is able to achieve a strong acceptance rate of 49.08% for multi-line Ansible task suggestions. With our findings we provide insights into the effectiveness of small, dedicated models in a domain-specific context.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。