基于因果效应估计的领域自适应模型压缩
Model Compression for Domain Adaptation through Causal Effect Estimation
摘要：
最近自然语言处理系统的预测质量的改进通常依赖于模型参数数量的大幅增加。这导致了压缩这种模型的各种尝试，但是现有的方法没有考虑各种模型组件的预测能力或压缩模型的可推广性的差异。为了理解模型压缩和分布外泛化之间的联系，我们定义了压缩语言表示模型的任务，使得它们在领域自适应设置中表现最佳。我们选择从因果关系的角度来解决这个问题，试图估计模型组件（如单层）对模型预测的平均治疗效果（ATE）。我们提出的ATE引导的模型压缩方案（AMoC），产生许多模型候选人，不同的模型组件被删除。然后，我们通过逐步回归模型选择最佳候选者，该模型利用ATE来预测目标域上的预期性能。在三个文本分类和序列标记任务中，AMoC在数十个域对上的表现优于强基线。
英文摘要：
Recent improvements in the predictive quality of natural language processing systems are often dependent on a substantial increase in the number of model parameters. This has led to various attempts of compressing such models, but existing methods have not considered the differences in the predictive power of various model components or in the generalizability of the compressed models. To understand the connection between model compression and out-of-distribution generalization, we define the task of compressing language representation models such that they perform best in a domain adaptation setting. We choose to address this problem from a causal perspective, attempting to estimate the average treatment effect (ATE) of a model component, such as a single layer, on the model's predictions. Our proposed ATE-guided Model Compression scheme (AMoC), generates many model candidates, differing by the model components that were removed. Then, we select the best candidate through a stepwise regression model that utilizes the ATE to predict the expected performance on the target domain. AMoC outperforms strong baselines on dozens of domain pairs across three text classification and sequence tagging tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
