发表机构
Independent Researcher（独立研究者）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对候选顺序影响决策评分的问题，提出候选无关块因果注意力，阻断跨候选信息流并重置位置，在多个骨干上降低排列敏感性且保持决策质量。
AI 中文摘要
决策模型通常对编码在单个序列中的一组大小可变的候选动作进行评分。这一设置对于生成系统内的System 1组件日益相关，在这些系统中，候选可能在不同运行中以不同顺序被提出或排序。标准的因果交叉编码具有表现力，但它可能使候选的得分依赖于序列化顺序，而非潜在的决策问题。我们引入了候选无关的块因果注意力，它在共享上下文和每个候选内部保持因果计算，同时阻断跨候选信息流并重置候选位置。我们将该架构与标准因果注意力及互补的不变基线在Gemma 3 1B、Qwen3 1.7B和Qwen3 4B骨干网络上进行了比较。候选无关注意力在保持竞争性决策质量的同时，持续降低了排列敏感性；消融实验表明，候选隔离是该效应的主要来源，而位置重置则完善了预期的对称性。一项更大的Qwen3-4B研究进一步考察了所提架构在显著更多训练数据下的行为。代码可在项目仓库中获取，Qwen3-4B模型工件可在Hugging Face上获取。
英文摘要
Decision models often score a variable-sized set of candidate actions encoded in a single sequence. This setting is increasingly relevant for System 1 components inside generative systems, where candidates may be proposed or ordered differently across runs. Standard causal cross-encoding is expressive, but it can make a candidate's score depend on serialization order rather than on the underlying decision problem. We introduce candidate-independent block-causal attention, which preserves causal computation within the shared context and each candidate while blocking cross-candidate information flow and resetting candidate positions. We compare this architecture with standard causal attention and complementary invariant baselines across Gemma 3 1B, Qwen3 1.7B, and Qwen3 4B backbones. Candidate-independent attention consistently reduces permutation sensitivity while retaining competitive decision quality; ablations indicate that candidate isolation is the primary source of the effect, with position resetting completing the intended symmetry. A larger Qwen3-4B study further examines the behavior of the proposed architecture with substantially more training data. Code is available at the \href{https://github.com/guyAmit/ci-decision-models}{\textcolor{blue}{project repository}}, and the \href{https://huggingface.co/Guy-Amit/qwen3-4b-ci-decision-4096-poc}{\textcolor{blue}{Qwen3-4B model artifact}} is available on Hugging Face.
CommentsTechnical Report, will not be submitted to a conference
