揭开黑箱:多模态基础模型中情感建模的初步洞察
Understanding Affective Adaptation in Multimodal Foundation Models: Emergent Functional Specialization
AI总结:
本研究揭示多模态基础模型中情感建模的关键机制,发现前馈门控投影模块对情感理解和生成至关重要,通过参数高效调整实现高性能。
摘要:
理解大规模基础模型中情感是如何被表示的仍然是一个开放性问题,尤其是在多模态情感设置中。尽管近期情感模型在实证表现上很强,但支持情感理解和生成的内部架构机制仍然理解不足。在本工作中,我们对多模态基础模型中的情感建模进行了系统性的机理研究。在多个架构、训练策略和情感任务中,我们分析了以情感为导向的监督如何重塑内部模型参数。我们的结果一致揭示出一个清晰而稳健的模式:情感适应主要不集中在注意力模块,而是局部化到前馈门控投影(gate_proj)。通过受控模块转移、针对性单模块适应和破坏性消融,我们进一步证明了gate_proj在情感理解和生成中是足够、高效且必要的。值得注意的是,仅调整约24.5%的参数(由AffectGPT调整的参数),我们的方法在八个情感任务上实现了96.6%的平均性能,突显了显著的参数效率。共同,这些发现提供了实证证据,表明基础模型中的情感能力是由前馈门控机制结构上调节的,并将gate_proj识别为情感建模的核心架构位置。
英文摘要:
Despite rapid progress in multimodal affective foundation models, how affective capabilities emerge within their internal architectures remains poorly understood. A critical open question is whether affective fine-tuning induces diffuse changes across the model or organizes computation into functionally specialized pathways. We systematically investigate this question through a broad module-level analysis of 13 affective model instances spanning nine model designs, multiple scales, tasks, and training paradigms, complemented by controlled functional analyses on representative models. We find that affective adaptation exhibits a consistent yet non-exclusive functional organization. Under matched trainable-parameter budgets, adapting only the feed-forward network (FFN) consistently outperforms adapting only the attention modules across all evaluated settings and, on average, nearly matches the performance obtained by tuning all major Transformer projections, identifying the FFN as a particularly efficient adaptation substrate. More strikingly, although the gate, up, and down projections exhibit comparable standalone adaptation capacity, their learned functional contributions become differentiated after joint optimization. Module recovery and targeted interventions identify \texttt{gate\_proj} as a particularly prominent pathway, while checkpoint analysis shows that this differentiation develops over the course of training. We characterize this phenomenon as emergent functional specialization: distinct pathway-level roles are not fully explained by standalone adaptation capacity, but arise through joint affective adaptation. Building on this finding, Gate-Focused Efficient Tuning (GET) retains 96.2-98.0\% of the performance obtained by tuning all major Transformer projections while using only 19.3-24.5\% as many trainable parameters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。