重排问题：当正确变为错误时在大语言模型中
The Realignment Problem: When Right becomes Wrong in LLMs
AI总结：
本文提出TRACE框架，通过结构化优化解决LLM在动态标注政策下的重排问题，无需新标注即可提升对齐效果。
摘要：
大语言模型（LLM）的后期对齐依赖于大规模人工标注，受政策变化影响，静态对齐变得脆弱。现有修复方法需重新标注，带来系统性挑战。本文提出TRACE框架，通过三阶段流程：（1）基于对齐冲突分类偏好对；（2）通过双层优化计算对齐影响评分；（3）使用混合目标更新模型。实验显示在Qwen2.5-7B、Gemma-2-9B和Llama-3.1-8B上，TRACE在合成基准和PKU-SafeRLHF数据集上有效，不降低通用性。
英文摘要：
Post-training alignment of large language models (LLMs) relies on large-scale human annotations guided by policy specifications that change over time. Cultural shifts, value reinterpretations, and regulatory or industrial updates make static alignment increasingly brittle. As policies evolve, deployed models can diverge from current alignment objectives, creating an Alignment-Reality Gap that is difficult to audit or correct. Existing remediation typically requires re-annotation under revised guidelines, which introduces systematic challenges, including guideline ambiguity, annotator interpretation drift, and reduced consistency at scale. We introduce TRACE (Triage and Re-align by Alignment Conflict Evaluation), a framework that transforms realignment into a structured optimization problem over existing data without requiring fresh human annotation. Leveraging a stronger model as a proxy judge, TRACE operates via a three-stage pipeline: (1) triaging preference pairs into inversion, suppression, or retention categories based on alignment conflicts; (2) computing an alignment impact score via bi-level optimization to prioritize high-leverage samples; and (3) executing updates using a hybrid objective that combines relational losses (e.g., IPO) for preference inversion and punitive losses (e.g., NPO) for response suppression. Experiments on Qwen2.5-7B, Gemma-2-9B, and Llama-3.1-8B demonstrate robust realignment on synthetic benchmarks and the PKU-SafeRLHF dataset without degrading general utility. This work provides a scalable approach for LLM realignment under evolving data annotation policies and alignment guidelines. We release our code: https://respailab.github.io/TRACE/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
