探索与表征用于嵌入式系统开发和调试的大型语言模型
Exploring and Characterizing Large Language Models For Embedded System Development and Debugging
AI总结：
本文开发了开源硬件在环框架评估LLMs在嵌入式系统开发中的能力，发现GPT-4具备跨领域推理优势，并提出人机协作工作流显著提升了不同经验开发者的生产力与成功率。
摘要：
大型语言模型（LLMs）在生成代码方面展现出了卓越的能力，然而，对于需要软硬件跨领域知识的嵌入式系统软件开发能力，尚未得到研究。在本文中，我们开发了一个可扩展的开源硬件在环框架，以系统性地评估领先的LLMs（GPT-3.5、GPT-4、PaLM 2），从而评估它们在嵌入式系统开发方面的能力和局限性。通过研究我们观察到，即使这些工具未能生成可运行的代码，它们也能持续生成关于嵌入式设计任务的有用推理。我们利用这一发现来研究人类程序员如何与这些工具交互，并开发了一种基于人机协作的软件工程工作流，用于构建嵌入式系统。我们用于验证LLM生成程序的评估平台使用传感器-执行器对进行物理评估。我们通过N=450次实验比较了所有三个模型，令人惊讶地发现，GPT-4尤其展现出了卓越的跨领域理解和推理能力，在某些情况下，仅凭单个提示就能生成完全正确的程序。在N=50次试验中，GPT-4有66%的时间生成了功能正常的I2C接口。GPT-4还生成了寄存器级驱动程序、LoRa通信代码，以及针对nRF52程序的上下文特定功耗优化，使电流降低了740倍以上，降至12.2uA。我们还表征了这些模型在嵌入式系统开发中使用LLMs的局限性，以开发一种可推广的人机协作工作流。我们与15名用户（包括新手和专家程序员）评估了我们的工作流。我们发现，我们的工作流提高了所有用户的生产力，并将构建LoRa环境传感器的成功率从25%提高到100%，包括对于零硬件或C/C++经验的用户。
英文摘要：
Large language models (LLMs) have shown remarkable abilities to generate code, however their ability to develop software for embedded systems, which requires cross-domain knowledge of hardware and software has not been studied. In this paper we develop an extensible, open source hardware-in-the-loop framework to systematically evaluate leading LLMs (GPT-3.5, GPT-4, PaLM 2) to assess their capabilities and limitations for embedded system development. We observe through our study that even when these tools fail to produce working code, they consistently generate helpful reasoning about embedded design tasks. We leverage this finding to study how human programmers interact with these tools, and develop an human-AI based software engineering workflow for building embedded systems. Our evaluation platform for verifying LLM generated programs uses sensor actuator pairs for physical evaluation. We compare all three models with N=450 experiments and find surprisingly that GPT-4 especially shows an exceptional level of cross-domain understanding and reasoning, in some cases generating fully correct programs from a single prompt. In N=50 trials, GPT-4 produces functional I2C interfaces 66% of the time. GPT-4 also produces register-level drivers, code for LoRa communication, and context-specific power optimizations for an nRF52 program resulting in over 740x current reduction to 12.2uA. We also characterize the models' limitations to develop a generalizable human-AI workflow for using LLMs in embedded system development. We evaluate our workflow with 15 users including novice and expert programmers. We find that our workflow improves productivity for all users and increases the success rate for building a LoRa environmental sensor from 25% to 100%, including for users with zero hardware or C/C++ experience.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
