Nowadays both commercial and open-source academic LLM have become the mainstream models of NLP. However, there is still a lack of research on LLM consistency, meaning that throughout the various stages of LLM research and deployment, its internal parameters and capabilities should remain unchanged. This issue exists in both the industrial and academic sectors. The solution to this problem is often time-consuming and labor-intensive, and there is also an additional cost of secondary deployment, resulting in economic and time losses. To fill this gap, we build an LLM consistency task dataset and design several baselines. Additionally, we choose models of diverse scales for the main experiments. Specifically, in the LightGBM experiment, we used traditional NLG metrics (i.e., ROUGE, BLEU, METEOR) as the features needed for model training. The final result exceeds the manual evaluation and GPT3.5 as well as other models in the main experiment, achieving the best performance. In the end, we use the best performing LightGBM model as the base model to build the evaluation tool, which can effectively assist in the deployment of business models. Our code and tool demo are available at https://github.com/heavenhellchen/Consistency.git
Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMs
School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);Nanyang Technological University(南洋理工大学);University of Leicester(莱斯特大学);Kuaishou Technology(快手科技)
Explaining Black-box Language Models with Knowledge Probing Systems: A Post-hoc Explanation Perspective
School of Computer and Information Technology, Shanxi University, China(山西大学计算机与信息学院);Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, Shanxi University, China(教育部计算智能与中文信息处理重点实验室);Institute for Infocomm Research, A*Star, Singapore(A*Star信息与通信研究所)
Department of School of Engineering, Shenzhen MSU-BIT University, Shenzhen, China, 518000(深圳MSU-BIT大学工程学院部门);MSU-BIT-SMBU Joint Research Center of Applied Mathematics, Shenzhen MSU-BIT University, Shenzhen, China, 518000(应用数学联合研究中心)
Perspectives on Cross-Lingual Consistency in LLMs for Medical Questions
Johannes Gutenberg University Mainz(美因茨约翰内斯·古腾堡大学);University of Colorado Boulder(科罗拉多大学博尔德分校);University of Marburg(马尔堡大学);Universidad Iberoamericana(伊比利亚美洲大学)
Explaining Black-box Model Predictions via Two-level Nested Feature Attributions with Consistency Property
STAIR Lab, Chiba Institute of Technology(千叶技术学院STAIR实验室);School of Mathematics and Statistics, The University of Melbourne(墨尔本大学数学与统计学学院);ZOZO Research(ZOZO研究)