arXiv 2606.163682026-06-16cs.CLcs.LG

通过语义约束验证评估LLM个性化

Evaluating LLM Personalization via Semantic Constraint Verification

University of New South Wales（新南威尔士大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；The Technology Innovation Institute（技术创新研究所）；The Hong Kong University of Science and Technology（香港科技大学）

Xuran Li, Guanqin Zhang, Imran Razzak, Hakim Hacid, Eleanna Kafeza, Hao Xue, Flora D. Salim

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出NLICV框架，利用自然语言推理模型将句子映射到真值条件集，验证个性化约束，将LLM行为分为四类，与人类标注高度一致，并大幅降低延迟和成本。

摘要：

当前大型语言模型（LLM）个性化的评估范式严重依赖于脆弱的表面匹配指标或计算成本高昂的LLM作为评判者的协议，两者都缺乏可解释性。为了解决这些局限性，我们引入了自然语言推理约束验证（NLICV），这是一个可扩展的、语义不变的框架，它将句子含义映射到真值条件集，通过自然语言推理（NLI）模型验证个性化约束。超越二元评分，NLICV将LLM行为分为四种不同模式：个性化、泛化、谄媚和失败。大量实验表明，NLICV与人工标注高度一致，同时大幅降低了与LLM评判者相关的延迟和令牌成本（高达2100倍推理加速）。最后，通过基于消融的程序，NLICV精确定位驱动约束验证的准确句子，为其评估提供忠实、可理解的证据。

英文摘要：

Current evaluation paradigms for Large Language Model (LLM) personalization rely heavily on brittle surface-matching metrics or computationally expensive LLM-as-a-judge protocols, both of which lack interpretability. To address these limitations, we introduce Natural Language Inference Constraint Verification (NLICV), a scalable, semantically invariant framework that maps sentence meanings to truth-condition sets to verify personalization constraints via a Natural Language Inference (NLI) model. Moving beyond binary scoring, NLICV categorizes LLM behaviors into four distinct modes: personalization, generalization, sycophancy, and failure. Extensive experiments demonstrate that NLICV aligns closely with human annotations while drastically reducing the latency and token costs associated with LLM judges (up to 2100 inference speedup). Finally, through an ablation-based procedure, NLICV pinpoints the exact sentences driving the constraint verification, yielding faithful, understandable evidence for its evaluations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。