arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.08097cs.SEcs.AIcs.CY
收藏

应用评论中的公平性问题:一项关于基于AI的移动应用的研究

Fairness Concerns in App Reviews: A Study on AI-based Mobile Apps

School of Computer Science, Wuhan University(武汉大学计算机学院);RMIT University(皇家墨尔本理工大学);University of Melbourne(墨尔本大学);Deakin University(迪肯大学);Monash University(莫纳什大学)

Ali Rezaei Nasab, Maedeh Dashti, Mojtaba Shahin, Mansooreh Zahedi, Hourieh Khalajzadeh, Chetan Arora, Peng Liang

AI总结:

本研究通过构建公平性评论数据集并训练机器学习模型,从约950万条基于AI的移动应用评论中识别出约9.2万条公平性评论,进而归纳出六类公平性问题和六种根本原因。

摘要:

公平性是软件系统中必须解决的社会技术问题之一。考虑到移动软件应用(apps)在全球各类人群中的普及程度,具有不公平行为和结果的移动应用可能影响全球人口的很大一部分,其影响可能超过任何其他类型的软件系统。用户在移动应用评论中表达了广泛的社会技术问题。本研究旨在调查移动应用评论中提出的公平性问题。我们的研究聚焦于基于AI的移动应用评论,因为基于AI的移动应用中出现不公平行为和结果的可能性可能高于非基于AI的应用。为此,我们首先人工构建了一个真实标注数据集,包括1,132条公平性评论和1,473条非公平性评论。利用该真实标注数据集,我们开发并评估了一组机器学习和深度学习模型,用于区分公平性评论与非公平性评论。我们的实验表明,表现最佳的模型能够以94%的精确率检测公平性评论。随后,我们将表现最佳的模型应用于从108个基于AI的应用中收集的约950万条评论,并识别出约9.2万条公平性评论。接下来,对9.2万条公平性评论应用K-means聚类技术,并进行人工分析,识别出六种不同类型的公平性问题(例如,“在不同平台和设备上获得不同质量的功能和服务”以及“在处理用户生成内容时缺乏透明度和公平性”)。最后,对2,248条应用所有者对公平性评论的回复进行人工分析,确定了应用所有者报告用于解释公平性问题的六种根本原因(例如,“版权问题”)。

英文摘要:

Fairness is one of the socio-technical concerns that must be addressed in software systems. Considering the popularity of mobile software applications (apps) among a wide range of individuals worldwide, mobile apps with unfair behaviors and outcomes can affect a significant proportion of the global population, potentially more than any other type of software system. Users express a wide range of socio-technical concerns in mobile app reviews. This research aims to investigate fairness concerns raised in mobile app reviews. Our research focuses on AI-based mobile app reviews as the chance of unfair behaviors and outcomes in AI-based mobile apps may be higher than in non-AI-based apps. To this end, we first manually constructed a ground-truth dataset, including 1,132 fairness and 1,473 non-fairness reviews. Leveraging the ground-truth dataset, we developed and evaluated a set of machine learning and deep learning models that distinguish fairness reviews from non-fairness reviews. Our experiments show that our best-performing model can detect fairness reviews with a precision of 94%. We then applied the best-performing model on approximately 9.5M reviews collected from 108 AI-based apps and identified around 92K fairness reviews. Next, applying the K-means clustering technique to the 92K fairness reviews, followed by manual analysis, led to the identification of six distinct types of fairness concerns (e.g., 'receiving different quality of features and services in different platforms and devices' and 'lack of transparency and fairness in dealing with user-generated content'). Finally, the manual analysis of 2,248 app owners' responses to the fairness reviews identified six root causes (e.g., 'copyright issues') that app owners report to justify fairness concerns.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑