arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.17411cs.CL
收藏

一致性至关重要:从黑盒视角探究 LLM 的一致性

Consistency Matters: Explore LLMs Consistency From a Black-Box Perspective

Fufangchen Zhao, Guoqiang Jin, Jiaheng Huang, Rui Zhao, Fei Tan

AI总结:

针对 LLM 研究与部署中的一致性问题,本文构建数据集与基线,并提出以传统 NLG 指标为特征的 LightGBM 黑盒评估方法,性能优于人工评估和 GPT3.5。

摘要:

如今,商业和开源学术 LLM 都已成为 NLP 的主流模型。然而,目前仍缺乏针对 LLM 一致性的研究;所谓一致性,是指在 LLM 研究与部署的各个阶段,其内部参数和能力应保持不变。这一问题在工业界和学术界均存在。解决该问题通常耗时耗力,还会产生二次部署的额外成本,造成经济和时间损失。为填补这一空白,我们构建了一个 LLM 一致性任务数据集,并设计了若干基线。此外,我们选择不同规模的模型进行主实验。具体而言,在 LightGBM 实验中,我们使用传统 NLG 指标(即 ROUGE、BLEU、METEOR)作为模型训练所需的特征。最终结果超过了人工评估以及主实验中的 GPT3.5 和其他模型,取得了最佳性能。最后,我们将性能最佳的 LightGBM 模型作为基础模型构建评估工具,该工具可有效辅助业务模型的部署。我们的代码和工具演示可在 https://github.com/heavenhellchen/Consistency.git 获取。

英文摘要:

Nowadays both commercial and open-source academic LLM have become the mainstream models of NLP. However, there is still a lack of research on LLM consistency, meaning that throughout the various stages of LLM research and deployment, its internal parameters and capabilities should remain unchanged. This issue exists in both the industrial and academic sectors. The solution to this problem is often time-consuming and labor-intensive, and there is also an additional cost of secondary deployment, resulting in economic and time losses. To fill this gap, we build an LLM consistency task dataset and design several baselines. Additionally, we choose models of diverse scales for the main experiments. Specifically, in the LightGBM experiment, we used traditional NLG metrics (i.e., ROUGE, BLEU, METEOR) as the features needed for model training. The final result exceeds the manual evaluation and GPT3.5 as well as other models in the main experiment, achieving the best performance. In the end, we use the best performing LightGBM model as the base model to build the evaluation tool, which can effectively assist in the deployment of business models. Our code and tool demo are available at https://github.com/heavenhellchen/Consistency.git

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑