基于感知质量的标注者标签不确定性下的模型训练
Perceptual Quality-based Model Training under Annotator Label Uncertainty
AI总结：
针对标注者标签不确定性导致模型性能下降的问题，提出基于感知质量的训练框架，通过选取低质量样本并生成多标签，缓解泛化与预测不确定性退化。
摘要：
标注者在数据标注过程中会表现出分歧，这可称为标注者标签不确定性。标注者标签不确定性表现为标注质量的差异。使用每个样本仅有一条低质量标注进行训练会导致模型可靠性下降。在本工作中，我们首先从模型的泛化能力和预测不确定性两方面考察标注者标签不确定性的影响。我们观察到，当存在低质量噪声标签时，模型的泛化能力和预测不确定性会下降。同时，我们对现有不确定性估计算法的评估表明，它们无法有效应对标注者标签不确定性。为缓解性能下降，先前的方法表明，使用从多个独立标注者收集的标签训练模型可以增强泛化能力。然而，这些方法需要大量的标注。因此，我们提出了一种新颖的基于感知质量的模型训练框架，以客观地为模型训练生成多个标签，从而在避免大量标注的同时增强可靠性。具体来说，我们首先根据视觉信号的统计规律性，选取感知质量得分较低的一个样本子集。然后，我们为该子集中的每个样本分配去聚合标签，以获得具有多个标签的训练集。我们的实验和分析表明，使用所提出的框架进行训练可缓解由标注者标签不确定性引起的泛化能力和预测不确定性的下降。
英文摘要：
Annotators exhibit disagreement during data labeling, which can be termed as annotator label uncertainty. Annotator label uncertainty manifests in variations of labeling quality. Training with a single low-quality annotation per sample induces model reliability degradations. In this work, we first examine the effects of annotator label uncertainty in terms of the model's generalizability and prediction uncertainty. We observe that the model's generalizability and prediction uncertainty degrade with the presence of low-quality noisy labels. Meanwhile, our evaluation of existing uncertainty estimation algorithms indicates their incapability in response to annotator label uncertainty. To mitigate performance degradation, prior methods show that training models with labels collected from multiple independent annotators can enhance generalizability. However, they require massive annotations. Hence, we introduce a novel perceptual quality-based model training framework to objectively generate multiple labels for model training to enhance reliability, while avoiding massive annotations. Specifically, we first select a subset of samples with low perceptual quality scores ranked by statistical regularities of visual signals. We then assign de-aggregated labels to each sample in this subset to obtain a training set with multiple labels. Our experiments and analysis demonstrate that training with the proposed framework alleviates the degradation of generalizability and prediction uncertainty caused by annotator label uncertainty.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
