高效多任务推理:基于 Gromov-Wasserstein 特征对齐的模型合并
Efficient Multi-Task Inferencing: Model Merging with Gromov-Wasserstein Feature Alignment
AI总结:
针对教育自动评分中多模型部署成本高的问题,提出 GW-SMM,利用 Gromov-Wasserstein 距离对齐特征分布并合并共享层,在保持评分性能的同时减少一半存储。
摘要:
学生回答的自动评分可提升教育效率,但为每个任务部署独立神经网络会增加存储需求、维护工作量和冗余计算。为应对这些挑战,本文提出 Gromov-Wasserstein Scoring Model Merging(GW-SMM)方法,该方法利用 Gromov-Wasserstein 距离衡量特征分布相似性,并据此合并模型。该方法首先使用各个独立模型从学生回答中提取特征,同时捕获题目特定上下文和独特的学习表示。随后,Gromov-Wasserstein 距离量化这些特征分布之间的相似性,识别最适合合并的模型。成对距离最小的模型通常以两个或三个为一组,仅通过合并分类头之前的共享层进行合并。该策略形成统一的特征提取器,同时保留用于题目特定评分的独立分类头。我们将所提方法与人类专家知识以及基于 GPT-o1 的合并方法进行了验证。GW-SMM 始终优于两者,取得了更高的 micro F1 分数、macro F1 分数、精确匹配准确率和逐标签准确率。与基于 GPT-o1 的合并相比,micro F1 和逐标签准确率的提升具有统计显著性(p=0.04,p=0.01)。此外,GW-SMM 将存储需求降低了一半,且未明显损失准确率,证明其在具备计算效率的同时拥有可靠的评分性能。
英文摘要:
Automatic scoring of student responses enhances efficiency in education, but deploying a separate neural network for each task increases storage demands, maintenance efforts, and redundant computations. To address these challenges, this paper introduces the Gromov-Wasserstein Scoring Model Merging (GW-SMM) method, which merges models based on feature distribution similarities measured via the Gromov-Wasserstein distance. Our approach begins by extracting features from student responses using individual models, capturing both item-specific context and unique learned representations. The Gromov-Wasserstein distance then quantifies the similarity between these feature distributions, identifying the most compatible models for merging. Models exhibiting the smallest pairwise distances, typically in pairs or trios, are merged by combining only the shared layers preceding the classification head. This strategy results in a unified feature extractor while preserving separate classification heads for item-specific scoring. We validated our approach against human expert knowledge and a GPT-o1-based merging method. GW-SMM consistently outperformed both, achieving a higher micro F1 score, macro F1 score, exact match accuracy, and per-label accuracy. The improvements in micro F1 and per-label accuracy were statistically significant compared to GPT-o1-based merging (p=0.04, p=0.01). Additionally, GW-SMM reduced storage requirements by half without compromising much accuracy, demonstrating its computational efficiency alongside reliable scoring performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。