TOWER: 用于评估复杂指令遵循能力的树结构组织加权方法
TOWER: Tree Organized Weighting for Evaluating Complex Instructions
AI总结：
针对现有大模型复杂指令遵循评估未区分指令各部分重要性的问题，提出融入人类重要性判断的树结构评估指标TOWER，验证了树结构表示的人类一致性，并发布InFoBench树标注与评估代码。
摘要：
评估大语言模型（LLM）遵循人类编写的复杂指令的能力，对其在实际应用中的部署至关重要。尽管Chatbot Arena等基准使用人类评审员评估模型性能，但这类方法资源消耗大且耗时。使用LLM作为评审的替代方法（如AlpacaEval、MT Bench、WildBench和InFoBench）有所改进，但仍未体现出复杂指令的某些方面比其他方面更需要被遵循的特性。为弥补这一缺口，我们提出了一种新型评估指标\textsc{TOWER}，将人类评审的重要性权重纳入复杂指令遵循能力的评估中。我们发现，人类标注员对这些复杂指令的树结构表示的认同度，几乎与他们对其他人类标注员的认同度相当。我们发布了InFoBench数据集的树结构标注及对应的评估代码，以推动未来研究。
英文摘要：
Evaluating the ability of large language models (LLMs) to follow complex human-written instructions is essential for their deployment in real-world applications. While benchmarks like Chatbot Arena use human judges to assess model performance, they are resource-intensive and time-consuming. Alternative methods using LLMs as judges, such as AlpacaEval, MT Bench, WildBench, and InFoBench offer improvements but still do not capture that certain complex instruction aspects are more important than others to follow. To address this gap, we propose a novel evaluation metric, \textsc{TOWER}, that incorporates human-judged importance into the assessment of complex instruction following. We show that human annotators agree with tree-based representations of these complex instructions nearly as much as they agree with other human annotators. We release tree-based annotations of the InFoBench dataset and the corresponding evaluation code to facilitate future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
