Tensor Train 低秩近似（TT-LoRA）：通过加速 LLM 普及 AI
Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs
AI总结：
提出 TT-LoRA 参数高效微调方法，以优化的 tensor train 分解替代 Adapters 和 LoRA 结构，在保持下游性能的同时提升 LLM 压缩率并降低推理开销。
摘要：
近年来，大语言模型（LLM）在问答、情感分析、文本摘要和机器翻译等广泛的自然语言处理（NLP）任务中展现出显著能力。然而，LLM 不断增长的复杂性需要巨大的计算资源，阻碍了这些模型更广泛的研究与应用。为解决这一问题，研究者开发了多种参数高效微调策略，例如低秩近似（LoRA）和 Adapters。尽管这些方法具有潜力，但它们往往在可压缩性方面存在局限。具体而言，随着现代大规模 LLM 中可训练参数数量增加，LoRA 难以有效扩展。此外，利用 tensor train 分解的 Low-Rank Economic Tensor-Train Adaptation（LoRETTA）尚未达到在资源有限条件下微调超大规模模型所需的压缩水平。本文提出 Tensor Train Low-Rank Approximation（TT-LoRA），这是一种新颖的参数高效微调（PEFT）方法，通过优化的 tensor train（TT）分解集成扩展了 LoRETTA。通过去除 Adapters 和传统的基于 LoRA 的结构，TT-LoRA 在不损害下游任务性能的情况下实现更高的模型压缩，同时降低推理延迟和计算开销。我们进行了详尽的参数搜索以建立基准，突出模型压缩与性能之间的权衡。结果表明，该方法可显著压缩 LLM，同时保持与更大模型相当的性能，从而促进其在资源受限平台上的部署。
英文摘要：
In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing (NLP) tasks, such as question-answering, sentiment analysis, text summarization, and machine translation. However, the ever-growing complexity of LLMs demands immense computational resources, hindering the broader research and application of these models. To address this, various parameter-efficient fine-tuning strategies, such as Low-Rank Approximation (LoRA) and Adapters, have been developed. Despite their potential, these methods often face limitations in compressibility. Specifically, LoRA struggles to scale effectively with the increasing number of trainable parameters in modern large scale LLMs. Additionally, Low-Rank Economic Tensor-Train Adaptation (LoRETTA), which utilizes tensor train decomposition, has not yet achieved the level of compression necessary for fine-tuning very large scale models with limited resources. This paper introduces Tensor Train Low-Rank Approximation (TT-LoRA), a novel parameter-efficient fine-tuning (PEFT) approach that extends LoRETTA with optimized tensor train (TT) decomposition integration. By eliminating Adapters and traditional LoRA-based structures, TT-LoRA achieves greater model compression without compromising downstream task performance, along with reduced inference latency and computational overhead. We conduct an exhaustive parameter search to establish benchmarks that highlight the trade-off between model compression and performance. Our results demonstrate significant compression of LLMs while maintaining comparable performance to larger models, facilitating their deployment on resource-constraint platforms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
