arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.05002cs.SE
收藏

LLM应用开发者面临挑战的实证研究

An Empirical Study on Challenges for LLM Application Developers

Xiang Chen, Chaoyang Gao, Chunyang Chen, Guangbei Zhang, Yong Liu

AI总结:

针对LLM应用开发给开发者带来的新挑战缺乏深入实证研究的问题,本研究通过爬取分析OpenAI开发者论坛的29057个问题并手动分析2364个样本,构建了挑战分类体系,为相关利益相关者提供了研究发现和可操作启示。

摘要:

近年来,大型语言模型(LLMs)发展迅速,对计算机视觉、自然语言处理和软件工程等多个领域产生了重大影响。以OpenAI的ChatGPT为代表的这些LLMs彻底改变了我们处理语言理解和生成任务的方式。然而,与传统软件开发实践相比,LLM开发在设计、实现和部署方面给AI开发者带来了新的挑战。这些挑战涉及不同领域(如提示词、API和插件),要求开发者掌握LLM应用开发特有的方法论和考量因素。尽管LLMs影响深远,但据我们所知,以往实证研究尚未深入探讨这些挑战。为填补这一空白,我们开展了首个全面研究以了解LLM开发者面临的挑战。具体而言,我们从一个热门OpenAI开发者论坛爬取并分析了29057个相关问题,首先考察其流行度和难度;手动分析2364个抽样问题后,构建了LLM开发者面临的挑战分类体系,并基于该体系为包括开发者和提供商(尤其是OpenAI机构)在内的LLM相关利益相关者总结了一系列研究发现和可操作的启示。

英文摘要:

In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as computer vision, natural language processing, and software engineering. These LLMs, exemplified by OpenAI's ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM application development. Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑