TACTICL：面向表格上下文学习模型的任务感知压缩框架
TACTICL: Task-Aware Compression of Tabular ICL Models
浏览论文内容
中文总结 AI 辅助
针对表格基础模型推理成本高、蒸馏后损失上下文适应性的问题，提出TACTICL框架，剪枝Transformer层并替换为轻量适配器，在47个基准数据集上可替换85%层且性能无显著下降，对数据偏移鲁棒。
中文摘要 AI 辅助
用于表格任务的基础模型虽性能强劲，但推理成本高昂。将模型蒸馏为特定任务架构可减小模型规模、降低计算需求，但会牺牲上下文适应性。本文提出TACTICL，一种面向表格上下文学习模型的自动化任务感知压缩框架，该框架联合剪枝Transformer层，并用在下游任务上训练的轻量适配器替代这些层，从而将上下文学习与权重内学习融合。在47个基准数据集上对TACTICL的研究表明，在给定下游任务上，我们可替换多达85%的层而不会出现显著性能下降。我们进一步证明，TACTICL对数据偏移保持鲁棒性，其上下文能力完好无损。总体而言，TACTICL通过结合特定任务适应与结构化压缩，为利用表格基础模型的深度冗余提供了一个鲁棒框架。我们提供代码：this https URL
英文摘要
The strong performance of foundation models for tabular tasks comes at substantial inference costs. Distilling models into task-specific architectures reduces model size and computational demands but also sacrifices in-context adaptability. Here we introduce TACTICL, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning. We study TACTICL on 47 benchmark datasets and show that we can substitute up to 85% of layers without substantial performance drop on a given downstream task. We further show that TACTICL maintains robustness to data shifts, leaving its in-context ability intact. Overall, TACTICL provides a robust framework for exploiting the depth-wise redundancy of tabular foundation models by combining task-specific adaptation and structured compression. We provide the code at: https://github.com/Hebog/tfm_compression
发表机构
- TU Dortmund University（多特蒙德工业大学）
- Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）
机构由 AI 辅助整理，请以论文原文为准。
