人格预测在AI招聘中有效性的外部稳定性审计框架
An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring
摘要：
自动招聘系统是所有高风险人工智能系统中发展最快的系统之一。其中包括算法人格测试，它利用心理测量测试的洞察力，并承诺根据求职者的简历或社交媒体资料来揭示预示未来成功的人格特征。我们询问这些系统的有效性，使用它们产生的输出的稳定性，注意到可靠性是必要的，但不是充分的，有效性的条件。我们的方法是（a）开发一种方法，用于对算法人格测试所做预测的稳定性进行外部审计，以及（b）在Humantic AI和Crystal两个系统的审计中实例化这种方法。至关重要的是，我们不是挑战或肯定心理测量测试中的假设--人格是一个有意义的、可测量的结构，人格特质是未来工作成功的指标--而是围绕测试算法人格测试供应商自己做出的基本假设来构建我们的方法。我们的主要贡献是开发了一个社会技术框架，用于审计算法系统的稳定性。这一贡献补充了一个开源的软件库，实现了审计的技术组件，并可用于进行类似的算法系统的稳定性审计。我们通过对两个现实世界的人格预测系统（即Humantic AI和Crystal）的审计来实例化我们的框架。我们的审计框架的应用表明，这两个系统显示出相当大的不稳定性方面的关键衡量，因此不能被视为有效的测试工具。
英文摘要：
Automated hiring systems are among the fastest-developing of all high-stakes AI systems. Among these are algorithmic personality tests that use insights from psychometric testing, and promise to surface personality traits indicative of future success based on job seekers' resumes or social media profiles. We interrogate the validity of such systems using stability of the outputs they produce, noting that reliability is a necessary, but not a sufficient, condition for validity. Our approach is to (a) develop a methodology for an external audit of stability of predictions made by algorithmic personality tests, and (b) instantiate this methodology in an audit of two systems, Humantic AI and Crystal. Crucially, rather than challenging or affirming the assumptions made in psychometric testing -- that personality is a meaningful and measurable construct, and that personality traits are indicative of future success on the job -- we frame our methodology around testing the underlying assumptions made by the vendors of the algorithmic personality tests themselves. Our main contribution is the development of a socio-technical framework for auditing the stability of algorithmic systems. This contribution is supplemented with an open-source software library that implements the technical components of the audit, and can be used to conduct similar stability audits of algorithmic systems. We instantiate our framework with the audit of two real-world personality prediction systems, namely Humantic AI and Crystal. The application of our audit framework demonstrates that both these systems show substantial instability with respect to key facets of measurement, and hence cannot be considered valid testing instruments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
