策略性人类智能体下的数据标注自动化：风险与潜在解决方案
Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutions
AI总结：
本文研究了在策略性人类智能体下使用模型标注样本重训练ML模型的长期影响，发现模型重训练会导致智能体获得正向决策概率增加而正向标签比例下降，并提出改进重训练过程以稳定动态，同时指出强制公平性约束长远未必有益。
摘要：
随着机器学习（ML）模型越来越多地被应用于社会领域以做出关于人类的重大决策，它们通常具有重塑数据分布的能力。人类作为策略性智能体，会不断调整自身行为以应对学习系统。由于群体动态变化，ML系统可能需要频繁更新以确保高性能。然而，在社会领域中获取高质量的人类标注样本极具挑战性，甚至不可行。解决此问题的常见做法是使用模型本身来标注无标签数据样本。本文研究了当ML模型在纳入人类策略响应的情况下，使用模型标注样本进行重训练时的长期影响。我们首先形式化了策略性智能体与模型之间的交互，然后分析了它们在这种动态交互下的演变。我们发现，随着模型被重训练，智能体越来越有可能获得正向决策，而具有正向标签的智能体比例可能随时间推移而下降。因此，我们提出了一种改进的重训练过程来稳定这种动态。最后，我们考察了这些重训练过程如何影响算法公平性，并发现在每一轮强制执行常见的公平性约束从长远来看未必能使弱势群体受益。在（半）合成和真实数据上的实验验证了理论发现。
英文摘要：
As machine learning (ML) models are increasingly used in social domains to make consequential decisions about humans, they often have the power to reshape data distributions. Humans, as strategic agents, continuously adapt their behaviors in response to the learning system. As populations change dynamically, ML systems may need frequent updates to ensure high performance. However, acquiring high-quality human-annotated samples can be highly challenging and even infeasible in social domains. A common practice to address this issue is using the model itself to annotate unlabeled data samples. This paper investigates the long-term impacts when ML models are retrained with model-annotated samples when they incorporate human strategic responses. We first formalize the interactions between strategic agents and the model and then analyze how they evolve under such dynamic interactions. We find that agents are increasingly likely to receive positive decisions as the model gets retrained, whereas the proportion of agents with positive labels may decrease over time. We thus propose a refined retraining process to stabilize the dynamics. Last, we examine how algorithmic fairness can be affected by these retraining processes and find that enforcing common fairness constraints at every round may not benefit the disadvantaged group in the long run. Experiments on (semi-)synthetic and real data validate the theoretical findings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
