关于排行榜运营（LBOps）的工作流与坏味：基础模型排行榜的探索性研究
On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards
AI总结：
本研究探索了基础模型排行榜的运营工作流，从五个来源收集了1,045个排行榜，识别出五种工作流模式、领域模型及八种排行榜坏味，以提升FM选择与比较的透明度和问责制。
摘要：
基础模型（FM），例如作为大规模机器学习（ML）模型的大型语言模型（LLM），在代码补全、代码理解和软件开发等各种下游软件工程（SE）任务中展现出了卓越的适应性。因此，FM排行榜已成为SE团队比较和选择最适合其特定产品与目的的第三方FM的重要工具。然而，缺乏标准化的FM评估与比较指南威胁到了FM排行榜的透明度，并限制了利益相关者有效选择FM的能力。作为应对这一挑战的第一步，我们的研究重点是了解这些FM排行榜在真实场景中的运作方式（即“排行榜运营”），并识别潜在的陷阱和改进领域（即“排行榜坏味”）。为此，我们从五个不同来源收集了多达1,045个FM排行榜：GitHub、Hugging Face Spaces、Papers With Code、电子表格和独立平台，以检查其文档并与排行榜运营者直接沟通以了解其工作流。通过卡片分类和协商一致，我们识别出五种不同的工作流模式，并开发了一个领域模型来捕捉这些工作流中的关键组件及其交互。随后，我们识别出LBOps中八种独特的排行榜坏味。通过缓解这些坏味，SE团队可以提高当前LBOps实践中的透明度、问责制和协作，从而为FM的比较与选择培育一个更稳健、更负责任的生态系统。
英文摘要：
Foundation models (FM), such as large language models (LLMs), which are large-scale machine learning (ML) models, have demonstrated remarkable adaptability in various downstream software engineering (SE) tasks, such as code completion, code understanding, and software development. As a result, FM leaderboards have become essential tools for SE teams to compare and select the best third-party FMs for their specific products and purposes. However, the lack of standardized guidelines for FM evaluation and comparison threatens the transparency of FM leaderboards and limits stakeholders' ability to perform effective FM selection. As a first step towards addressing this challenge, our research focuses on understanding how these FM leaderboards operate in real-world scenarios ("leaderboard operations") and identifying potential pitfalls and areas for improvement ("leaderboard smells"). In this regard, we collect up to 1,045 FM leaderboards from five different sources: GitHub, Hugging Face Spaces, Papers With Code, spreadsheet and independent platform, to examine their documentation and engage in direct communication with leaderboard operators to understand their workflows. Through card sorting and negotiated agreement, we identify five distinct workflow patterns and develop a domain model that captures the key components and their interactions within these workflows. We then identify eight unique types of leaderboard smells in LBOps. By mitigating these smells, SE teams can improve transparency, accountability, and collaboration in current LBOps practices, fostering a more robust and responsible ecosystem for FM comparison and selection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
