arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.09150cs.CL
收藏

确保GPT-3可靠

Prompting GPT-3 To Be Reliable

University of Maryland(马里兰大学);Microsoft(微软)

Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, Lijuan Wang

摘要:

大语言模型(LLM)通过Few-Shot提示显示出令人印象深刻的能力。商业化的API(如OpenAI GPT-3)进一步增加了它们在现实世界语言应用程序中的使用。然而,如何提高GPT-3的可靠性的关键问题仍然是探索不足。虽然可靠性是一个宽泛且定义模糊的术语,但我们将可靠性分解为四个主要方面,这些方面与现有的ML安全框架相对应,并且被公认为非常重要:可推广性,社会偏见,校准和真实性。我们的核心贡献是建立简单有效的提示,提高GPT-3的可靠性,因为它:1)概括了分布,2)平衡人口分布,并使用自然语言指令来减少社会偏见,3)校准输出概率,4)更新LLM的事实知识和推理链。通过适当的提示,GPT-3在所有这些方面都比小规模的监督模型更可靠。我们发布所有处理过的数据集、评估脚本和模型预测。我们的系统性实证研究不仅对LLM的提示可靠性提出了新的见解,更重要的是,我们的提示策略可以帮助从业者更可靠地使用LLM,如GPT-3。

英文摘要:

Large language models (LLMs) show impressive abilities via few-shot prompting. Commercialized APIs such as OpenAI GPT-3 further increase their use in real-world language applications. However, the crucial problem of how to improve the reliability of GPT-3 is still under-explored. While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality. Our core contribution is to establish simple and effective prompts that improve GPT-3's reliability as it: 1) generalizes out-of-distribution, 2) balances demographic distribution and uses natural language instructions to reduce social biases, 3) calibrates output probabilities, and 4) updates the LLM's factual knowledge and reasoning chains. With appropriate prompts, GPT-3 is more reliable than smaller-scale supervised models on all these facets. We release all processed datasets, evaluation scripts, and model predictions. Our systematic empirical study not only sheds new insights on the reliability of prompting LLMs, but more importantly, our prompting strategies can help practitioners more reliably use LLMs like GPT-3.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑