arXiv 2603.222912026-03-04cs.CL

评估大型语言模型对尼泊尔性与生殖健康问题的回应

Evaluating Large Language Models' Responses to Sexual and Reproductive Health Queries in Nepali

Nepal Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息学研究所）；Partnership for Sustainable Development Nepal（尼泊尔可持续发展伙伴关系）；Visible Impact（可见影响）

Medha Sharma, Supriya Khadka, Udit Chandra Aryal, Bishnu Hari Bhatta, Bijayan Bhattarai, Santosh Dahal, Kamal Gautam, Pushpa Joshi, Saugat Kafle, Shristi Khadka… 展开作者

Medha Sharma, Supriya Khadka, Udit Chandra Aryal, Bishnu Hari Bhatta, Bijayan Bhattarai, Santosh Dahal, Kamal Gautam, Pushpa Joshi, Saugat Kafle, Shristi Khadka, Shushila Khadka, Binod Lamichhane, Shilpa Lamichhane, Anusha Parajuli, Sabina Pokharel, Suvekshya Sitaula, Neha Verma, Bishesh Khanal

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LEAF框架，评估大型语言模型在尼泊尔性与生殖健康问题上的准确性、语言、可用性及安全性，发现仅35.1%的回应符合标准，揭示了当前模型的局限性。

摘要：

随着大型语言模型（LLMs）融入日常生活，它们被越来越多地用于个人查询，包括性与生殖健康（SRH），使用户能够匿名交流而无需担心评判。然而，当前的评估方法主要关注准确性，通常针对高资源语言中的客观查询，缺乏评估可用性和安全性标准，尤其在低资源语言和敏感领域如SRH中。本文介绍了LLM评估框架（LEAF），该框架涵盖多个标准：准确性、语言、可用性缺口（包括相关性、充分性和文化适当性）以及安全性缺口（安全性、敏感性和保密性）。利用LEAF框架，我们评估了来自超过9000名用户、14000个SRH查询的尼泊尔语回应。回应由SRH专家根据框架进行人工标注。结果显示，仅有35.1%的回应被判定为“合适”，即准确、充分且无重大可用性或安全性缺口。洞察包括不同ChatGPT版本之间的性能差异，如相似的准确性但不同的可用性和安全性方面。此评估突显了当前LLMs的显著局限，并强调了改进的必要性。LEAF框架适用于各种领域和语言，特别是在可用性和安全性至关重要的情况下，为更好地处理敏感话题提供了路径。

英文摘要：

As Large Language Models (LLMs) become integrated into daily life, they are increasingly used for personal queries, including Sexual and Reproductive Health (SRH), allowing users to chat anonymously without fear of judgment. However, current evaluation methods primarily focus on accuracy, often for objective queries in high-resource languages, and lack criteria to assess usability and safety, especially for low-resource languages and culturally sensitive domains like SRH. This paper introduces LLM Evaluation Framework (LEAF), that conducts assessments across multiple criteria: accuracy, language, usability gaps (including relevance, adequacy, and cultural appropriateness), and safety gaps (safety, sensitivity, and confidentiality). Using the LEAF framework, we assessed 14K SRH queries in Nepali from over 9K users. Responses were manually annotated by SRH experts according to the framework. Results revealed that only 35.1% of the responses were "proper", meaning they were accurate, adequate and had no major usability or safety related gaps. Insights include differences in performance between ChatGPT versions, such as similar accuracy but varying usability and safety aspects. This evaluation highlights significant limitations of current LLMs and underscores the need for improvement. The LEAF Framework is adaptable across domains and languages, particularly where usability and safety are critical, offering a pathway to better address sensitive topics.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。