arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.06276cs.SE
收藏

利用大语言模型进行故事点估计

Story Point Estimation Using Large Language Models

Rochester Institute of Technology(罗切斯特理工学院)

Pranam Prakash Shetty, Adarsh Balakrishnan, Mengqiao Xu, Xiaoyin Xi, Zhe Yu

AI总结:

研究探讨了大语言模型在无训练数据或少量训练数据下对故事点的预测能力,并比较了比较判断与直接标注对模型性能的影响。

摘要:

本研究探讨了利用大语言模型(LLMs)进行故事点估计的应用。故事点是无单位的项目特定努力估计,帮助Scrum团队成员预测冲刺中计划完成的产品待办事项。为促进此过程,机器学习模型,尤其是深度神经网络,已被用于基于每个事项的标题和描述预测故事点。然而,此类机器学习模型需要足够数量的训练数据(具有地面真实故事点标注的人类开发者)才能获得良好的预测性能。这促使我们探索LLMs是否能够(RQ1)在无训练数据或(RQ2)仅少量训练数据下预测故事点。对四个LLMs在16个软件项目上的实证结果表明,无任何训练数据(零样本提示)下,LLMs预测故事点的性能优于监督深度学习模型(训练数据为80%的数据)。此外,一项近期研究探讨了使用比较判断(在给定的一对事项中,哪一项需要更多努力来实现)代替直接标注故事点以减少开发者的认知负担。因此,本研究也探讨(RQ3)比较判断是否比故事点更容易被LLMs预测,以及(RQ4)比较判断是否能作为LLMs的少量示例以提高其对故事点的预测性能。实证结果表明,LLMs预测比较判断并不比直接估计故事点更容易,但比较判断可以作为少量示例以提高LLMs的预测性能以及人工标注的故事点。

英文摘要:

This study investigates the use of large language models (LLMs) for story point estimation. Story points are unitless, project-specific effort estimates that help developers on the scrum team forecast which product backlog items they plan to complete in a sprint. To facilitate this process, machine learning models, especially deep neural networks, have been applied to predict the story points based on the title and description of each item. However, such machine learning models require sufficient amounts of training data (with ground truth story points annotated by human developers) from the same software project to achieve decent prediction performance. This motivated us to explore whether LLMs are capable of (RQ1) predicting story points without training data or (RQ2) with only a few training data points. Our empirical results with four LLMs on 16 software projects show that, without any training data (zero-shot prompting), LLMs can predict story points better than supervised deep learning models trained on 80% of the data. The prediction performance of LLMs can be further improved with a few training examples (few-shot prompting). In addition, a recent study explored the use of comparative judgments (between a given pair of items which one requires more effort to implement) instead of directly annotating the story points to reduce the cognitive burden on developers. Therefore, this study also explores (RQ3) whether comparative judgments are easier to predict than story points for LLMs and (RQ4) whether comparative judgments can serve as few-shot examples for LLMs to improve their predictions of story points. Empirical results suggest that it is not easier for LLMs to predict comparative judgments than to directly estimate the story points, but comparative judgments can serve as few-shot examples to improve the LLMs' prediction performance as well as the human-annotated story points.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑