迈向兼具韧性与效率的大语言模型：效率、性能与对抗鲁棒性的比较研究
Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness
AI总结：
本研究针对注意力高效型大语言模型的对抗鲁棒性研究不足的问题，构建框架对比三类不同架构LLM的效率、性能与鲁棒性，发现简化架构模型可在三者间实现更优平衡，为资源受限且需抗对抗攻击的场景提供参考。
摘要：
随着大语言模型（LLMs）实际应用需求的不断增长，众多注意力高效型模型被开发出来以平衡性能与计算成本，但这类模型的对抗鲁棒性仍有待充分探究。本研究设计了一个框架，用于考察LLMs在效率、性能与对抗鲁棒性之间的权衡关系，并针对三种复杂度与效率水平各异的主流模型——Transformer++、门控线性注意力（Gated Linear Attention, GLA）Transformer以及无矩阵乘法语言模型（MatMul-Free LM）——开展了大量实验，实验采用GLUE和AdvGLUE数据集，其中AdvGLUE数据集在GLUE基础上扩展了旨在挑战模型鲁棒性的对抗样本。研究结果表明，尽管GLA Transformer与MatMul-Free LM在GLUE任务上的准确率略低，但在不同攻击强度下的AdvGLUE任务中，它们相比Transformer++展现出更高的效率，且鲁棒性更优或相当。这些发现凸显了简化架构在实现效率、性能与对抗鲁棒性之间出色平衡方面的潜力，为资源受限且对抗攻击韧性要求高的应用场景提供了宝贵参考。
英文摘要：
With the increasing demand for practical applications of Large Language Models (LLMs), many attention-efficient models have been developed to balance performance and computational cost. However, the adversarial robustness of these models remains under-explored. In this work, we design a framework to investigate the trade-off between efficiency, performance, and adversarial robustness of LLMs and conduct extensive experiments on three prominent models with varying levels of complexity and efficiency -- Transformer++, Gated Linear Attention (GLA) Transformer, and MatMul-Free LM -- utilizing the GLUE and AdvGLUE datasets. The AdvGLUE dataset extends the GLUE dataset with adversarial samples designed to challenge model robustness. Our results show that while the GLA Transformer and MatMul-Free LM achieve slightly lower accuracy on GLUE tasks, they demonstrate higher efficiency and either superior or comparative robustness on AdvGLUE tasks compared to Transformer++ across different attack levels. These findings highlight the potential of simplified architectures to achieve a compelling balance between efficiency, performance, and adversarial robustness, offering valuable insights for applications where resource constraints and resilience to adversarial attacks are critical.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
