arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.15979cs.SEcs.AI
收藏

OLAF:迈向面向经验软件工程的鲁棒LLM注释框架

OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering

Missouri University of Science and Technology(密苏里科技大学);University of Maryland Baltimore County(马里兰大学巴尔的摩县分校)

Mia Mohammad Imran, Tarannum Shaila Zaman

AI总结:

本文提出OLAF框架,旨在提升基于LLM的注释在经验软件工程中的透明性和可重复性。

摘要:

大型语言模型(LLMs)在经验软件工程(ESE)中被越来越多地用于自动化或协助注释任务,如标记提交、问题和定性制品。然而,此类注释的可靠性和可重复性仍缺乏深入研究。现有研究往往缺乏标准化的可靠性、校准和漂移测量标准,并且经常省略关键的配置细节。我们认为,基于LLM的注释应被视为一种测量过程,而不是纯粹的自动化活动。在本文中,我们提出了\textbf{基于LLM注释框架的操作化(OLAF)},一个概念框架,组织了关键构念:\textit{可靠性、校准、漂移、共识、聚合}和\textit{透明性}。本文旨在推动方法论讨论和未来经验研究,以实现更透明和可重复的基于LLM的注释在软件工程研究中。

英文摘要:

Large Language Models (LLMs) are increasingly used in empirical software engineering (ESE) to automate or assist annotation tasks such as labeling commits, issues, and qualitative artifacts. Yet the reliability and reproducibility of such annotations remain underexplored. Existing studies often lack standardized measures for reliability, calibration, and drift, and frequently omit essential configuration details. We argue that LLM-based annotation should be treated as a measurement process rather than a purely automated activity. In this position paper, we outline the \textbf{Operationalization for LLM-based Annotation Framework (OLAF)}, a conceptual framework that organizes key constructs: \textit{reliability, calibration, drift, consensus, aggregation}, and \textit{transparency}. The paper aims to motivate methodological discussion and future empirical work toward more transparent and reproducible LLM-based annotation in software engineering research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑