Akpans是捣蛋还是款待：揭示助手系统中的有益偏见
Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems
AI总结：
本文研究信息寻求AI助手系统有益性的计算测量方法，通过收集人类标注开发自动评估模型，并利用有益性水平衡量系统公平性，实验发现现有系统对不同发展水平国家存在偏见。
摘要：
信息寻求AI助手系统旨在及时回答用户的知识查询。然而，信息寻求助手系统在人类感知上的有益性及其公平性影响均未得到充分探索。本文研究了有益性的计算测量方法。我们收集了关于对话响应有益性的人类标注，开发了用于自动评估有益性的模型，随后提出利用对话系统对不同用户查询的有益性水平来衡量对话系统的公平性。在包括ChatGPT在内的最先进对话系统上，于三种信息寻求场景下进行的实验表明，现有系统在回答关于高度发达国家概念的问题时往往比欠发达国家更有帮助，揭示了当前信息寻求助手系统潜在的公平性问题。
英文摘要：
Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner. However, both the human-perceived helpfulness of information-seeking assistant systems and its fairness implication are under-explored. In this paper, we study computational measurements of helpfulness. We collect human annotations on the helpfulness of dialogue responses, develop models for automatic helpfulness evaluation, and then propose to use the helpfulness level of a dialogue system towards different user queries to gauge the fairness of a dialogue system. Experiments with state-of-the-art dialogue systems, including ChatGPT, under three information-seeking scenarios reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries than less-developed countries, uncovering potential fairness concerns underlying the current information-seeking assistant systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
