IDGen：面向大语言模型评估的项目区分度诱导提示生成方法
IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation
AI总结：
受教育评估中的项目区分度理论启发，本研究提出IDGen提示合成框架，结合自校正机制与区分度、难度预测模型，生成更具挑战性和区分度的LLM评估数据，并将发布含3000余条提示的数据集。
摘要：
随着大语言模型（LLM）在处理复杂任务方面的能力日益增强，评估集必须跟上这些发展步伐，以确保其仍具备足够的区分度。项目区分度（Item Discrimination，ID）理论被广泛应用于教育评估领域，用于衡量单个测试题目区分高水平和低水平受试者的能力。受该理论启发，我们提出了一种项目区分度诱导的提示合成框架，用于大语言模型评估，确保评估集能够根据模型能力持续更新与优化。\n我们的数据合成框架同时兼顾广度与特异性，既能生成可全面评估大语言模型能力的提示，又能揭示模型之间有意义的性能差异，从而有效区分它们在不同任务和领域的相对优缺点。为生成高质量数据，我们在泛化框架中引入了自校正机制，并开发了两个模型分别预测提示的区分度和难度分数，为数据合成框架提供支撑，也为评估数据合成研究贡献了有价值的工具。\n我们使用生成的数据评估了5个SOTA模型。我们的数据平均得分为51.92，方差为10.06。相比之下，已有研究（即SELF-INSTRUCT和WizardLM）的平均得分超过67，方差低于3.2。结果表明，与已有研究相比，我们的框架生成的数据更具挑战性和区分度。我们将发布一个包含3000余条精心设计的提示的数据集，以推动大语言模型评估研究。
英文摘要：
As Large Language Models (LLMs) grow increasingly adept at managing complex tasks, the evaluation set must keep pace with these advancements to ensure it remains sufficiently discriminative. Item Discrimination (ID) theory, which is widely used in educational assessment, measures the ability of individual test items to differentiate between high and low performers. Inspired by this theory, we propose an ID-induced prompt synthesis framework for evaluating LLMs to ensure the evaluation set can continually update and refine according to model abilities. Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs while revealing meaningful performance differences between models, allowing for effective discrimination of their relative strengths and weaknesses across various tasks and domains. To produce high-quality data, we incorporate a self-correct mechanism into our generalization framework, and develop two models to predict prompt discrimination and difficulty score to facilitate our data synthesis framework, contributing valuable tools to evaluation data synthesis research. We apply our generated data to evaluate five SOTA models. Our data achieves an average score of 51.92, accompanied by a variance of 10.06. By contrast, previous works (i.e., SELF-INSTRUCT and WizardLM) obtain an average score exceeding 67, with a variance below 3.2. The results demonstrate that the data generated by our framework is more challenging and discriminative compared to previous works. We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
