思维链作为镜像：评估大语言模型与人类偏好之间结构化推理的对齐
Chain-of-Thought as a Lens: Evaluating Structured Reasoning Alignment between Human Preferences and Large Language Models
AI总结：
本文提出一种量化评估大语言模型多步结构化推理与人类偏好对齐的方法，引入对齐分数指标，通过构建基于语义熵的矩阵比较模型生成的思维链与人类偏好参考，发现对齐分数在2步推理时达到峰值，支持其作为诊断信号。
摘要：
本文主要展示了一种量化评估大语言模型多步结构化推理与人类偏好对齐的方法。我们引入对齐分数，一种语义层面的度量标准，通过在中间步骤上构建基于语义熵的矩阵，将模型生成的思维链轨迹与人类偏好的参考进行比较。我们的分析显示，对齐分数在不同模型和跳跃深度下跟踪任务准确性，并在2步推理时达到峰值。实证结果进一步表明，更大推理深度的不一致主要由对齐错误如主题转移和冗余推理驱动。将思维链采样视为从推理路径分布中抽样，我们实证展示了对齐分数与准确性、可读性和连贯性之间强而一致的相关性，支持其作为诊断信号的使用。代码已提供。
英文摘要：
This paper primarily demonstrates a method to quantitatively assess the alignment between multi-step, structured reasoning in large language models and human preferences. We introduce the Alignment Score, a semantic-level metric that compares a model-produced chain of thought traces with a human-preferred reference by constructing semantic-entropy-based matrices over intermediate steps and measuring their divergence. Our analysis shows that Alignment Score tracks task accuracy across models and hop depths, and peaks at 2-hop reasoning. Empirical results further indicate that misalignment at greater reasoning depths is driven mainly by alignment errors such as thematic shift and redundant reasoning. Viewing chain sampling as drawing from a distribution over reasoning paths, we empirically demonstrate a strong and consistent correlation between Alignment Score and accuracy, readability, and coherence, supporting its use as a diagnostic signal. The code is available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
