SkillVerse：基于树评估的LLM评估与增强
SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation
AI总结：
本文提出SkillVerse，一个无监督树结构诊断框架，通过LLM评判和树状图组织模型响应，提供细粒度能力评估，并在少样本选择和弱点预测任务中显著提升性能。
摘要：
随着语言模型演进以处理复杂多面的任务，其评估也必须适应以捕捉这种复杂性。对模型能力的细粒度、特定技能的理解能够赋能研究人员制定明智的模型开发计划。本文提出SkillVerse，一个无监督的树结构诊断框架，用于理解模型在特定能力上的熟练度。以LLM作为评判者，SkillVerse首先对模型响应进行评判，然后将其组织成称为树状图的层次结构。鉴于任意粒度级别的熟练度，SkillVerse能灵活地产生对现代大型模型行为的洞察。我们还展示了其在两个下游任务中的有效性：1）使用树搜索算法选择更具信息量的少样本示例，将模型上下文学习提升25%；2）以55%的成功率准确预测新模型弱点，比不使用SkillVerse高22%。
英文摘要：
As language models evolve to tackle complex, multifaceted tasks, their evaluation must adapt to capture this intricacy. A granular, skill-specific understanding of model capabilities can empower researchers to make informed model development plans. In this paper, we introduce SkillVerse, an unsupervised tree-structured diagnosis framework for understanding model proficiency in specific abilities. With LLM as a judge, SkillVerse first critiques the model responses, and then organizes them into a hierarchical structure termed dendrogram. Given proficiency at arbitrary levels of granularity, SkillVerse is flexible to produce insights of behaviors of modern large models. We also demonstrate its efficacy in two downstream tasks: 1) improving model in-context learning by 25% using a tree-search algorithm to select more informative few-shot demonstrations, and 2) accurately predicting new model weaknesses with a 55% success rate, 22% higher than without SkillVerse.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
