SignDPO：基于骨骼的无手势符号翻译的多级直接偏好优化
SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation
AI总结：
SignDPO通过多级直接偏好优化提升骨骼基签名语翻译的对齐效果，通过自引导机制和自动语言级偏好生成器解决语义漂移问题，实验表明其在多个基准测试中优于现有无手势符号方法。
摘要：
我们提出了SignDPO，一种新颖的多级直接偏好优化（DPO）框架，旨在增强基于骨骼的签名语翻译对齐。尽管当前基于骨骼的模型通过最大似然估计取得了显著进展，但主要受限于模仿范式，缺乏对签名语言细粒度时空细微差别的判别敏感性，常导致语义漂移。为解决此问题，SignDPO将优化目标从简单的序列模仿转向跨空间、时间和语言维度的结构化偏好对齐。我们的框架包含三个关键设计。首先，我们引入分层扰动策略，自动构建空间和时间非偏好样本，粒度包括全局和局部。其次，我们提出自引导机制，利用解码器交叉注意力分数识别并扰动语义显著的骨骼区域，迫使模型区分真实签名信号与结构扭曲。第三，我们通过微调专用扰动模型建立自动语言级偏好生成器，捕捉复杂输出级失败模式，无需人工标注。在三个广泛采用的基准测试CSL-Daily、How2Sign和OpenASL上进行的大量实验表明，SignDPO在多个基准测试中一致优于最先进的无手势符号方法，甚至能与传统有手势符号方法竞争。我们的结果表明，多级偏好对齐是弥合高熵骨骼轨迹与离散语言语义之间差距的强大范式。
英文摘要：
We present SignDPO, a novel multi-level Direct Preference Optimisation (DPO) framework designed to enhance the alignment of skeleton-based Sign Language Translation. While current skeleton-based models have made significant progress using Maximum Likelihood Estimation, they are primarily constrained by an imitation-based paradigm that lacks discriminative sensitivity to the fine-grained spatio-temporal nuances of sign language, often leading to semantic drift. To address this, SignDPO shifts the optimisation goal from simple sequence mimicry to structured preference alignment across spatial, temporal, and linguistic dimensions. Our framework involves three key designs. First, we introduce a hierarchical perturbation strategy to construct spatial and temporal non-preferred samples at both global and local granularities automatically. Second, we propose a self-guiding mechanism that leverages decoder cross-attention scores to identify and perturb semantically salient skeletal regions, forcing the model to distinguish genuine sign signals from structural distortions. Third, we establish an automated language-level preference generator by fine-tuning a dedicated perturbation model, capturing complex output-level failure modes without manual annotation. Extensive experiments on three widely adopted benchmarks, CSL-Daily, How2Sign, and OpenASL, demonstrate that SignDPO consistently outperforms state-of-the-art gloss-free methods and even rivals established gloss-based ones. Our results suggest that multi-level preference alignment is a powerful paradigm for bridging the gap between high-entropy skeletal trajectories and discrete linguistic semantics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
