语言模型将相关性编码为分层跨语言信号
LLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal
AI总结：
研究大语言模型中查询与文档相关性是否可线性解码，通过引导中等规模模型、提取激活并训练线性探针，发现相关性是深度依赖信号，多语言实验有部分跨语言可移植性，为基于LLM的相关性评估提供表征视角。
AI中文摘要：
大语言模型（LLMs）越来越多地用于信息检索（IR）管道作为相关性判断和重排器。但大多数分析仍以输出为中心，对模型内部相关性表示的洞察有限。本文研究查询-文档（q-d）相关性是否可从指令微调的LLMs中的残差流激活线性解码，该信号与生成的相关性判断如何比较，以及是否跨语言转移。使用TREC DL20和MIRACL评估集，用UMBRELA风格相关性判断提示引导中等规模LLMs（4-9B参数），从各层提取最后token激活并训练线性探针预测相关性标签。结果表明q-d相关性编码为深度依赖信号，早期层探针性能弱，中晚期最强，多语言实验表明有部分跨语言可移植性。这项工作为基于LLM的相关性评估提供了表征层面视角。
英文摘要：
Large language models (LLMs) are increasingly used in information retrieval (IR) pipelines as relevance judges and re-rankers. Yet most analyses remain output-centric, evaluating generated labels or scores while offering limited insight into how relevance is represented inside the model. In this work, we study whether query-document (q-d) relevance is linearly decodable from residual-stream activations in instruction-tuned LLMs, how this signal compares with generated relevance judgments, and whether it transfers across languages. Using the TREC DL20 and MIRACL evaluation collections, we guide medium-scale LLMs (4-9B parameters) with UMBRELA-style relevance judgment prompts, extract last-token activations from every transformer layer, and train linear probes to predict relevance labels. We compare probe predictions with generated judgments and use TREC DL20 to test whether probe-derived pseudo-labels preserve system rankings against human judgments. Our results suggest that q-d relevance is encoded as a depth-dependent signal: probe performance is weak in early layers and strongest in middle-to-late layers, indicating that relevance becomes more linearly accessible after contextual integration. Most importantly, in several models, validation-selected probes match or outperform generated judgments and better preserve system rankings, revealing a separation between internal relevance representation and external expression. Multilingual experiments suggest partial cross-language portability, although transfer remains weaker than within-language decoding. Overall, this work provides a representation-level perspective on LLM-based relevance assessment. Layer-wise probing can help diagnose where relevance emerges, when generated judgments fail to reflect internally available evidence, and how relevance representations vary across languages, datasets, and model families.
