基于白盒特征的大型语言模型自评估
Self-Evaluation of Large Language Model based on Glass-box Features
AI总结:
针对现有大型语言模型评估依赖外部评估器而忽视模型自身白盒特征的问题,提出利用 softmax 分布等白盒特征进行模型自评估,并在公共基准上验证了该方法的可行性。
摘要:
开源大型语言模型(LLMs)的激增加剧了对评估方法的迫切需求。现有工作主要依赖外部评估器,侧重于训练与提示策略。然而,模型感知的白盒特征这一关键维度却被忽视。本研究在自评估场景下探索白盒特征的效用,即应用 LLM 评估其自身输出。我们研究了多种白盒特征组,发现 softmax 分布可作为自评估的可靠质量指标。在公共基准上的实验结果验证了使用白盒特征进行 LLM 自评估的可行性。
英文摘要:
The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect, model-aware glass-box features, is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable quality indicator for self-evaluation. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。