工程化AI裁判系统
Engineering AI Judge Systems
AI总结：
针对FMware动态随机性导致AI裁判系统开发面临耗时、成本高和判断不准确的挑战，本文提出一种框架，通过案例研究验证其可将AI裁判系统准确率提升6.2%并显著减少开发工作量。
摘要：
AI裁判系统旨在自动评估由基础模型驱动的软件（即FMware）。由于FMware固有的动态性和随机性，AI裁判系统的开发需要独特的工程生命周期，并带来了新的挑战。本文基于我们在为FMware开发AI裁判系统的工业经验，讨论了这些挑战。这些挑战导致大量的时间消耗、成本增加和判断不准确。我们提出了一个框架来应对这些挑战，目标是提高开发高质量AI裁判系统的生产力。最后，我们通过一个针对提交信息生成FMware的裁判案例研究来评估我们的框架。使用我们的框架开发的AI裁判系统的判断准确率比未使用该框架开发的AI裁判系统高出多达6.2%，同时开发工作量显著减少。
英文摘要：
AI judge systems are designed to automatically evaluate Foundation Model-powered software (i.e., FMware). Due to the intrinsic dynamic and stochastic nature of FMware, the development of AI judge systems requires a unique engineering life cycle and presents new challenges. In this paper, we discuss the challenges based on our industrial experiences in developing AI judge systems for FMware. These challenges lead to substantial time consumption, cost and inaccurate judgments. We propose a framework that tackles the challenges with the goal of improving the productivity of developing high-quality AI judge systems. Finally, we evaluate our framework with a case study on judging a commit message generation FMware. The accuracy of the judgments made by the AI judge system developed with our framework outperforms those made by the AI judge system that is developed without our framework by up to 6.2%, with a significant reduction in development effort.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
