发表机构
Xidian University; Hong Kong University of Science and Technology (Guangzhou)（西安电子科技大学; 香港科技大学（广州））
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对在线策略蒸馏的成员审计需求，本文提出PAMA框架，引入TAG结合学生漂移等信号，在MATH数据集上AUC较基线提升14.6%-20.6%，实现更优的提示级成员审计。
AI 中文摘要
在线策略蒸馏（OPD）通过在学生模型自身生成的轨迹上，使学生模型的策略与教师模型的策略对齐，从而训练学生模型。在此过程中，学生策略会在蒸馏所用的提示上向教师策略靠拢。然而，这些提示通常是私密且成本高昂的，因此需要进行提示级别的成员审计。现有方法主要依赖基于似然的置信信号或检查点之间的学生策略漂移，但它们未捕捉到学生更新中由教师引导的方向。本文提出了一种专为在线策略蒸馏设计的新型审计框架——策略对齐成员审计（PAMA）。我们的关键观察是，成员提示直接促成教师引导的策略更新，而非成员提示仅通过跨提示泛化产生间接影响。基于这一方向轨迹，PAMA会测量学生更新是否朝着降低候选提示上的教师损失的方向移动。具体而言，我们引入教师对齐增益（TAG），以从模型输出中估计教师对齐的更新方向，并将其与学生漂移和不确定性对齐信号相结合，用于可靠的成员审计。我们在六个数据集和三个师生模型家族上对PAMA进行评估。在主要评估基准MATH上，PAMA的AUC值达到0.791至0.941，相较于最先进的基线方法，AUC提升了14.6%至20.6%。
英文摘要
On-policy distillation (OPD) trains a student model by aligning its policy with a teacher model on trajectories generated by the student model itself. Through this process, the student policy moves toward the teacher on the prompts used for distillation. However, these prompts are often private and costly, creating a need for prompt-level membership auditing. Existing methods mainly rely on likelihood-based confidence signals or student policy drift between checkpoints, but they do not capture the teacher-induced direction of the student update. In this paper, we propose Policy Alignment Membership Auditing (PAMA), a new auditing framework tailored for OPD. Our key observation is that a member prompt directly contributes to the teacher-guided policy update, while a non-member prompt only experiences indirect effects through cross-prompt generalization. Based on this directional trace, PAMA measures whether the student update moves toward reducing the teacher loss on a candidate prompt. Specifically, we introduce Teacher Alignment Gain (TAG) to estimate the teacher-aligned update direction from model outputs, and further combine it with student drift and uncertainty alignment signals for reliable membership auditing. We evaluate PAMA on six datasets and three teacher-student model families. On MATH, the primary evaluation benchmark, PAMA achieves AUC values of 0.791--0.941, improving AUC by 14.6--20.6% over state-of-the-art baselines.
Comments10 pages, 8 figures
