任务向量基:一种统一且可扩展的压缩任务算术框架
Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic
AI总结:
提出任务向量基框架,将T个任务向量压缩为M<T个基向量,支持加法、否定等算术操作,理论保证泛化与遗忘,实验表明压缩后性能优于或持平完整集合。
摘要:
任务算术通过任务向量上的线性操作表示下游任务,已成为跨不同设置迁移知识的简单而强大的范式。然而,维护大量任务向量在存储和计算方面带来了可扩展性挑战。我们提出了任务向量基框架,将$T$个任务向量压缩为$M < T$个基向量,同时保留任务算术的功能。通过将每个任务向量表示为基原子的结构化线性组合,我们的方法支持标准操作(如加法、否定)以及更高级的算术操作。该框架与任务算术中其他面向效率的改进正交,并可与之结合使用。我们提供了理论分析,表明基压缩保留了加法泛化保证,并实现了原则性的遗忘,误差界限取决于重建质量。实验上,我们提出的基构建方法始终优于启发式基构建基线,并且在某些情况下,在减少存储和计算需求的同时,甚至超越了完整任务向量集合在各种下游应用中的性能。代码可在以下网址获取:https://this URL。
英文摘要:
Task arithmetic, representing downstream tasks through linear operations on task vectors, has emerged as a simple yet powerful paradigm for transferring knowledge across diverse settings. However, maintaining a large collection of task vectors introduces scalability challenges in both storage and computation. We propose Task Vector Bases, a framework compressing $T$ task vectors into $M < T$ basis vectors while preserving the functionality of task arithmetic. By representing each task vector as a structured linear combination of basis atoms, our approach supports standard operations such as addition, negation, as well as more advanced arithmetic ones. The framework is orthogonal to other efficiency-oriented improvements in task arithmetic and can be used in combination with them. We provide theoretical analysis showing that basis compression retains addition generalization guarantees and enables principled unlearning, with error bounds depending on reconstruction quality. Empirically, our proposed basis construction methods consistently outperform heuristic basis construction baselines and, in some cases, even surpass the performance of full task vector collections across diverse downstream applications while reducing storage and computational requirements. The code is available at https://github.com/uiuctml/TaskVectorBasis.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。