追溯标注者思维：通过阅读过程增强偏好判断
Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processes
AI总结：
通过分析标注者的阅读行为，研究发现重读与标注一致性相关，揭示了标注过程中的认知机制。
摘要：
我们提出了一种标注方法，不仅捕捉标签，还捕捉标注者决策背后的阅读过程，例如他们关注文本的哪些部分、重读或快速浏览。使用该框架，我们对偏好标注任务进行了案例研究，创建了一个包含从鼠标跟踪获取的细粒度标注者阅读行为的数据集PreferRead。PreferRead使能够详细分析标注者在选择偏好之前在提示和两个候选响应之间导航的方式。我们发现，标注者在大约一半的试验中重读响应，大多数情况下是重读最终选择的选项，而很少重读提示。阅读行为也与标注结果显著相关：重读与更高的标注者间一致性相关，而长阅读路径和时间与较低的一致性相关。这些结果表明，阅读过程为理解复杂主观NLP任务中标注者可靠性、决策和分歧提供了补充认知维度。我们的代码和数据已公开可用。
英文摘要：
We propose an annotation approach that captures not only labels but also the reading process underlying annotators' decisions, e.g., what parts of the text they focus on, re-read or skim. Using this framework, we conduct a case study on the preference annotation task, creating a dataset PreferRead that contains fine-grained annotator reading behaviors obtained from mouse tracking. PreferRead enables detailed analysis of how annotators navigate between a prompt and two candidate responses before selecting their preference. We find that annotators re-read a response in roughly half of all trials, most often revisiting the option they ultimately choose, and rarely revisit the prompt. Reading behaviors are also significantly related to annotation outcomes: re-reading is associated with higher inter-annotator agreement, whereas long reading paths and times are associated with lower agreement. These results demonstrate that reading processes provide a complementary cognitive dimension for understanding annotator reliability, decision-making and disagreement in complex, subjective NLP tasks. Our code and data are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
