arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.17839cs.CLcs.AIcs.IR
收藏

Global-Liar:大语言模型在不同时间和地理区域下的事实性

Global-Liar: Factuality of LLMs over Time and Geographic Regions

New York University(纽约大学);New York University Abu Dhabi(纽约大学阿布扎比分校)

Shujaat Mirza, Bruno Coelho, Yuyuan Cui, Christina Pöpper, Damon McCoy

AI总结:

本研究构建地理时间均衡的Global-Liar数据集,评估GPT系列模型的事实性与偏差,发现模型更新未必提升性能、存在南北地域偏差,并揭示配置设置对事实性的影响,呼吁构建多元包容的训练评估体系以实现全球技术公平。

摘要:

人们日益依赖人工智能驱动的解决方案(尤其是GPT系列等大语言模型(LLM))进行信息检索,这凸显了对其事实性与公平性的迫切需求,在网络错误信息和虚假信息猖獗传播的背景下尤为如此。本研究评估了广泛使用的GPT模型(包括GPT-3.5和GPT-4)的事实准确性、稳定性和偏差,为人工智能介导的信息传播的可靠性与完整性提供支撑。\n 我们提出了Global-Liar数据集,该数据集在地理和时间表征上实现了独特的均衡,有助于更细致地评估大语言模型的偏差。分析显示,GPT模型的更新迭代并不总能带来性能提升。值得注意的是,3月版本的GPT-4比后续6月发布的版本具有更高的事实准确性。此外,研究还观察到一种令人担忧的偏差:模型更偏向全球北方地区的陈述,而非全球南方地区的陈述,这可能会加剧现有的信息不平等。非洲和中东等地区处于劣势,其事实准确性低得多。性能随时间的波动表明,模型更新可能无法始终平等地惠及所有地区。\n 本研究还揭示了大语言模型的各类配置设置(如二元决策强制、模型重跑和温度参数)对其事实性的影响。与允许“不确定”选项的模型相比,被限制为二元(是/否)选择的模型事实性会下降。在低温度设置下的单次推理,其可靠性可与多种配置下的多数投票相当。研究所得见解凸显了开展文化多元、地理包容的模型训练与评估的必要性,这一途径是实现全球技术公平、在全球范围内公平分配人工智能收益的关键。

英文摘要:

The increasing reliance on AI-driven solutions, particularly Large Language Models (LLMs) like the GPT series, for information retrieval highlights the critical need for their factuality and fairness, especially amidst the rampant spread of misinformation and disinformation online. Our study evaluates the factual accuracy, stability, and biases in widely adopted GPT models, including GPT-3.5 and GPT-4, contributing to reliability and integrity of AI-mediated information dissemination. We introduce 'Global-Liar,' a dataset uniquely balanced in terms of geographic and temporal representation, facilitating a more nuanced evaluation of LLM biases. Our analysis reveals that newer iterations of GPT models do not always equate to improved performance. Notably, the GPT-4 version from March demonstrates higher factual accuracy than its subsequent June release. Furthermore, a concerning bias is observed, privileging statements from the Global North over the Global South, thus potentially exacerbating existing informational inequities. Regions such as Africa and the Middle East are at a disadvantage, with much lower factual accuracy. The performance fluctuations over time suggest that model updates may not consistently benefit all regions equally. Our study also offers insights into the impact of various LLM configuration settings, such as binary decision forcing, model re-runs and temperature, on model's factuality. Models constrained to binary (true/false) choices exhibit reduced factuality compared to those allowing an 'unclear' option. Single inference at a low temperature setting matches the reliability of majority voting across various configurations. The insights gained highlight the need for culturally diverse and geographically inclusive model training and evaluation. This approach is key to achieving global equity in technology, distributing AI benefits fairly worldwide.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑