arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.00077cs.CLcs.LG
收藏

基于 LLM 的方法是否足以检测不公平的服务条款?

Are LLM-based methods good enough for detecting unfair terms of service?

DIGIT, Department of Electrical and Computer Engineering(DIGIT电子与计算机工程系);Aarhus University(奥胡斯大学)

Mirgita Frasheri, Arian Bakhtiarnia, Lukas Esterle, Alexandros Iosifidis

AI总结:

本文构建含12个问题的隐私政策数据集,评估开源与商业 LLM 检测不公平服务条款的能力,发现最佳模型为 ChatGPT4,但整体仅略优于随机,尚不足以大规模应用。

摘要:

世界各地的用户每天在与各种应用程序和网站交互时,都会签署无数的服务条款(ToS)。通常情况下,这些长达数十页的在线合同被只想立即获得所需服务的用户盲目签署。通常需要咨询法律团队的事情,如今已变成只需点击几下的日常活动,而用户可能在此过程中向无数在线实体/公司放弃自己的权利,例如在数据隐私方面。大型语言模型(LLM)擅长解析基于文本的长文档,因此有可能被采用,以帮助用户处理 ToS 及其底层隐私政策中可疑的条款。为研究现有模型在该任务上的效用,我们首先构建了一个数据集,其中包含 12 个问题,并将这些问题分别应用于从热门网站爬取的一组隐私政策。随后,我们围绕每个问题查询一系列开源及商业聊天机器人(例如 ChatGPT),并将其回答与给定的真实答案进行比较。结果表明,一些开源模型能够比某些商业模型提供更高的准确率。然而,最佳性能由一款商业聊天机器人(ChatGPT4)取得。总体而言,所有模型在该任务上的表现都仅略好于随机水平。因此,在它们能够被大规模用于此目的之前,其性能仍需得到显著提升。

英文摘要:

Countless terms of service (ToS) are being signed everyday by users all over the world while interacting with all kinds of apps and websites. More often than not, these online contracts spanning double-digit pages are signed blindly by users who simply want immediate access to the desired service. What would normally require a consultation with a legal team, has now become a mundane activity consisting of a few clicks where users potentially sign away their rights, for instance in terms of their data privacy, to countless online entities/companies. Large language models (LLMs) are good at parsing long text-based documents, and could potentially be adopted to help users when dealing with dubious clauses in ToS and their underlying privacy policies. To investigate the utility of existing models for this task, we first build a dataset consisting of 12 questions applied individually to a set of privacy policies crawled from popular websites. Thereafter, a series of open-source as well as commercial chatbots such as ChatGPT, are queried over each question, with the answers being compared to a given ground truth. Our results show that some open-source models are able to provide a higher accuracy compared to some commercial models. However, the best performance is recorded from a commercial chatbot (ChatGPT4). Overall, all models perform only slightly better than random at this task. Consequently, their performance needs to be significantly improved before they can be adopted at large for this purpose.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑