This study introduces the "Grade Score", a novel metric designed to evaluate the consistency and fairness of Large Language Models (LLMs) when used as multiple-choice judges with respect to order bias and choice consistency. The Grade Score combines Entropy, which measures order bias, and Mode Frequency, which assesses choice stability, offering insights into LLMs' reliability and impartiality. The study explores techniques such as prompt engineering and option sampling strategies to optimize the Grade Score, demonstrating their effectiveness in enhancing LLMs' performance. Results showcase varying performances among LLMs with respect to prompts and highlight the positive impact of including irrelevant options. The study also identifies an emergent behavior in instruction-following models, where they adapt to instructions targeting specific biases, demonstrating their adaptability. The Grade Score facilitates comparisons between LLMs and encourages ongoing research towards optimizing their decision-making processes, with potential implications for improving their reliability and fairness in various applications. All code is available on GitHub https://github.com/IoDmitri/GradeLab
Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Central China Normal University(华中师范大学);Shandong Science and Technology Press, Shandong Publishing Group(山东科学技术出版社,山东出版集团);Bielefeld University(比勒费尔德大学);South China Normal University(华南师范大学);Frankfurt Institute for Advanced Studies(法兰克福高等研究院)