arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.07398cs.CLcs.AIcs.SE
收藏

超越关键词:一种基于上下文的混合方法用于挖掘伦理关切相关的应用评论

Beyond Keywords: A Context-based Hybrid Approach to Mining Ethical Concern-related App Reviews

University of Calgary(卡尔加里大学)

Aakash Sorathiya, Gouri Ginde

AI总结:

本研究提出结合自然语言推理(NLI)和大型语言模型(LLM)的混合方法,从心理健康领域的应用评论中大规模提取伦理关切相关的隐私评论,克服了传统关键词方法的局限。

摘要:

随着移动应用在日常生活中的日益普及,围绕伦理的担忧显著增加。用户通常在应用评论中传达反馈、报告问题并建议新功能,经常强调安全性、隐私和问责制方面的担忧。纳入这些评论对于开发成功的产品至关重要。然而,与伦理关切相关的应用评论通常使用特定领域的语言,并且用更加多样的词汇来表达。因此,使得自动化提取伦理关切相关的应用评论成为一项具有挑战性且耗时的工作。本研究提出了一种基于自然语言处理(NLP)的新方法,该方法结合了自然语言推理(NLI,提供对语言细微差别的深刻理解)和一个仅包含解码器的(类似LLaMA的)大型语言模型(LLM),以大规模提取与伦理关切相关的应用评论。利用来自心理健康领域的43,647条应用评论,所提出的方法 1) 评估了四个NLI模型以提取潜在的隐私评论,并将特定领域隐私假设的结果与通用隐私假设的结果进行比较;2) 评估了四个LLM用于对应用评论进行隐私关切分类;3) 进一步使用最佳的NLI和LLM模型从数据集中提取新的隐私评论。结果表明,带有特定领域假设的DeBERTa-v3-base-mnli-fever-anli NLI模型产生了最佳性能,而Llama3.1-8B-Instruct LLM在应用评论分类中表现最佳。随后,使用NLI+LLM,额外提取了1,008条在先前研究中未被基于关键词的方法识别的新的隐私相关评论,从而证明了所提出方法的有效性。

英文摘要:

With the increasing proliferation of mobile applications in our everyday experiences, the concerns surrounding ethics have surged significantly. Users generally communicate their feedback, report issues, and suggest new functionalities in application (app) reviews, frequently emphasizing safety, privacy, and accountability concerns. Incorporating these reviews is essential to developing successful products. However, app reviews related to ethical concerns generally use domain-specific language and are expressed using a more varied vocabulary. Thus making automated ethical concern-related app review extraction a challenging and time-consuming effort. This study proposes a novel Natural Language Processing (NLP) based approach that combines Natural Language Inference (NLI), which provides a deep comprehension of language nuances, and a decoder-only (LLaMA-like) Large Language Model (LLM) to extract ethical concern-related app reviews at scale. Utilizing 43,647 app reviews from the mental health domain, the proposed methodology 1) Evaluates four NLI models to extract potential privacy reviews and compares the results of domain-specific privacy hypotheses with generic privacy hypotheses; 2) Evaluates four LLMs for classifying app reviews to privacy concerns; and 3) Uses the best NLI and LLM models further to extract new privacy reviews from the dataset. Results show that the DeBERTa-v3-base-mnli-fever-anli NLI model with domain-specific hypotheses yields the best performance, and Llama3.1-8B-Instruct LLM performs best in the classification of app reviews. Then, using NLI+LLM, an additional 1,008 new privacy-related reviews were extracted that were not identified through the keyword-based approach in previous research, thus demonstrating the effectiveness of the proposed approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑