Scaling-Aware Adapter for Structure-Grounded LLM Reasoning
Scaling-Aware Adapter for Structure-Grounded LLM Reasoning
AI总结：
提出Cuttlefish，一种统一多模态大语言模型，通过缩放感知适配器和几何基础适配器，自适应调整结构令牌数量并注入几何线索，以提升异构结构推理性能。
摘要：
大型语言模型（LLM）正在实现对2D和3D结构的推理，但现有方法仍然局限于特定模态，通常通过基于序列的标记化或固定长度的查询连接器来压缩结构输入。这种架构要么忽略了减轻结构幻觉所需的几何基础，要么施加了不灵活的模态融合瓶颈，同时过度压缩和次优分配结构令牌，从而阻碍了通用全原子推理的实现。我们引入了Cuttlefish，一种统一的多模态LLM，它将语言推理建立在几何线索上，同时根据结构复杂性缩放模态令牌。首先，缩放感知补丁利用指令条件门控机制在结构图上生成可变大小的补丁，根据结构复杂性自适应地缩放查询令牌预算，以缓解固定长度连接器的瓶颈。其次，几何基础适配器通过交叉注意力对模态嵌入进行细化，并将生成的模态令牌注入LLM，暴露明确的几何线索以减少结构幻觉。跨学科全原子基准的实验表明，Cuttlefish在异构结构基础推理中实现了优越的性能。代码：github.com/zihao-jing/Cuttlefish。
英文摘要：
Large language models (LLMs) are enabling reasoning over 2D and 3D structures, yet existing methods remain modality-specific and typically compress structural inputs through sequence-based tokenization or fixed-length query connectors. Such architectures either omit the geometric grounding requisite for mitigating structural hallucinations, or impose inflexible modality fusion bottlenecks that concurrently over-compress and suboptimally allocate structural tokens, thereby impeding the realization of generalized all-atom reasoning. We introduce Cuttlefish, a unified multimodal LLM that grounds language reasoning in geometric cues while scaling modality tokens with structural complexity. First, Scaling-Aware Patching leverages an instruction-conditioned gating mechanism to generate variable-size patches over structural graphs, adaptively scaling the query token budget with structural complexity to mitigate fixed-length connector bottlenecks. Second, Geometry Grounding Adapter refines these adaptive tokens via cross-attention to modality embeddings and injects the resulting modality tokens into the LLM, exposing explicit geometric cues to reduce structural hallucination. Experiments across interdisciplinary all-atom benchmarks demonstrate that Cuttlefish achieves superior performance in heterogeneous structure-grounded reasoning. Code: github.com/zihao-jing/Cuttlefish.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
