DIALEVAL: 大型语言模型指令遵循的自动类型理论评估
DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following
AI总结：
DIALEVAL通过双LLM代理实现复杂指令的自动类型理论评估，提升对话场景下的评估准确性与人类判断一致性。
摘要：
评估大型语言模型中的指令遵循需要将指令分解为可验证的要求并评估满足程度——这些任务目前依赖于人工标注和统一标准，这些标准与人类判断模式不一致。我们提出了DIALEVAL，一个使用双LLM代理的类型理论框架，用于自动化将指令分解为类型谓词并实现类型特定的满足语义。该框架在自动化提取过程中强制执行正式的原子性和独立性约束，然后应用不同的评估标准——语义等价用于内容谓词，精确度用于数值谓词——这与经验观察到的人类评估模式相匹配。通过历史感知的满足函数扩展到多轮对话，DIALEVAL使在对话上下文中进行评估成为可能，而单轮方法在此类场景中失效。验证显示其准确率为90.38%（比基线减少了26.45%的误差），并且在复杂指令上与人类判断有显著更强的相关性。
英文摘要：
Evaluating instruction following in Large Language Models requires decomposing instructions into verifiable requirements and assessing satisfaction--tasks currently dependent on manual annotation and uniform criteria that do not align with human judgment patterns. We present DIALEVAL, a type-theoretic framework using dual LLM agents to automate instruction decomposition into typed predicates and implement type-specific satisfaction semantics. The framework enforces formal atomicity and independence constraints during automated extraction, then applies differentiated evaluation criteria--semantic equivalence for content predicates, exact precision for numerical predicates--mirroring empirically observed human assessment patterns. Extended to multi-turn dialogues through history-aware satisfaction functions, DIALEVAL enables evaluation in conversational contexts where single-turn methods fail. Validation demonstrates 90.38% accuracy (26.45% error reduction over baselines) and substantially stronger correlation with human judgment for complex instructions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
