FLaG：用于 Token 聚合的频域潜在注意力门控池化

中文摘要 AI 辅助

Token 聚合将 token 级表示转换为固定维度的样本表示，但大多数池化方法仅在原始 token 空间中操作。我们引入频域潜在注意力门控池化（Frequency-Domain Latent-attention Gated Pooling，FLaG），这是一种可即插即用的聚合模块，在最终池化前将编码器输出重新表示到傅里叶域中。FLaG 通过串联实部和虚部表示非冗余的 rFFT 频谱，用可学习的潜在查询总结频谱 token，推导样本条件通道门，并重构调制后的 token 表示用于下游聚合。我们在基于 ESM2 的抗菌肽（Antimicrobial Peptide，AMP）活性预测、在 CIFAR-10 和 CIFAR-100 上的 ResNet18 图像分类，以及三个基于 RoBERTa 的语言任务中评估了相同的架构。FLaG 在四种 AMP 骨干-物种设置中取得了最佳的宏平均 Spearman 相关系数、RMSE 和 Recall@50，在 CIFAR-10 上取得了最高的 Top-1 准确率；在七个语言指标中的五个上也取得了最佳的平均结果，尽管在 STSBenchmark 上平均池化仍然最强。AMP 侧的机制分析显示，在大多数编码器层中对低频预测敏感，在最后一层中相对高频敏感性增加，且存在明显的肽特异性位置响应。残差门在保留低频主导能量分布的同时广泛放大频谱通道，而潜在交叉注意力表现出样本和物种特异性的频谱分配。总体而言，FLaG 在蛋白质、视觉和文本表示之间提供了可迁移的频域聚合偏差，其益处取决于骨干网络和下游任务。补充材料、源代码和数据可在该 https URL 和该 https URL 获取。

英文摘要

Token aggregation converts token-level representations into fixed-dimensional sample representations, but most pooling methods operate only in the original token space. We introduce Frequency-Domain Latent-attention Gated Pooling (FLaG), a plug-in aggregation module that re-expresses encoder outputs in the Fourier domain before final pooling. FLaG represents the nonredundant rFFT spectrum through concatenated real and imaginary components, summarizes spectral tokens with learnable latent queries, derives a sample-conditioned channel gate, and reconstructs modulated token representations for downstream aggregation. We evaluate the same architecture across ESM2-based antimicrobial peptide (AMP) activity prediction, ResNet18 image classification on CIFAR-10 and CIFAR-100, and three RoBERTa-based language tasks. FLaG achieves the best macro-averaged Spearman correlation coefficient, RMSE, and Recall@50 across four AMP backbone-species settings and the highest top-1 accuracy on CIFAR 10. It also achieves the best mean results on five of seven language metrics, although mean pooling remains strongest on STSBenchmark. AMP-side mechanistic analyses reveal low-frequency prediction sensitivity across most encoder layers, with increased relative high-frequency sensitivity in the final layer, and pronounced peptide-specific positional responses. The residual gate broadly amplifies spectral channels while preserving the low-frequency-dominated energy profile, whereas latent cross-attention exhibits sample- and species-specific spectral allocation. Overall, FLaG provides a transferable frequency-domain aggregation bias across protein, visual, and textual representations, with benefits that depend on the backbone and downstream task. Supplementary materials, source code, and data are available at https://www.healthinformaticslab.org/supp/ and https://github.com/Kewei2023/AMPCliff/tree/FLaG.

发表机构

• Jilin University（吉林大学）
• University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
• Greenwich High School（格林威治高中）
• BCPM Data Limited（BCPM数据有限公司）
• University of Cambridge（剑桥大学）

机构由 AI 辅助整理，请以论文原文为准。