LexInstructEval: 用于大语言模型的词汇指令遵循评估
LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models
AI总结：
本文提出LexInstructEval，通过形式化规则语法系统，系统生成多样化数据集并提供透明评估工具，以评估大语言模型对复杂词汇指令的遵循能力。
摘要：
大语言模型（LLMs）精确遵循复杂和精细的词汇指令的能力是其效用和可控性的核心。然而，评估这一能力仍是一个重大挑战。当前方法要么依赖主观且昂贵的人工评估，要么依赖自动LLM-as-judge系统，这些系统存在固有偏见和不可靠性。现有的程序化基准虽然客观，但往往缺乏测试复杂、组合性约束的表达能力。为了解决这些限制，我们引入LexInstructEval，一个新的基准和评估框架，用于细粒度词汇指令遵循。我们的框架基于一个正式、规则化的语法，将复杂指令分解为标准的<Procedure, Relation, Value>三元组。该语法使通过多阶段、人机协作的流程系统生成多样化数据集成为可能，并通过透明的程序化引擎实现客观验证。我们发布了我们的数据集和开源评估工具，以促进进一步研究LLM的可控性和可靠性。
英文摘要：
The ability of Large Language Models (LLMs) to precisely follow complex and fine-grained lexical instructions is a cornerstone of their utility and controllability. However, evaluating this capability remains a significant challenge. Current methods either rely on subjective and costly human evaluation or on automated LLM-as-a-judge systems, which suffer from inherent biases and unreliability. Existing programmatic benchmarks, while objective, often lack the expressiveness to test intricate, compositional constraints at a granular level. To address these limitations, we introduce LexInstructEval, a new benchmark and evaluation framework for fine-grained lexical instruction following. Our framework is built upon a formal, rule-based grammar that deconstructs complex instructions into a canonical <Procedure, Relation, Value> triplet. This grammar enables the systematic generation of a diverse dataset through a multi-stage, human-in-the-loop pipeline and facilitates objective verification via a transparent, programmatic engine. We release our dataset and open-source evaluation tools to facilitate further research into the controllability and reliability of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
