arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.07519cs.CLcs.AI
收藏

语言表征模型可以用打赌来思考吗?

Can Language Representation Models Think in Bets?

Information Sciences Institute(信息科学研究所);USC Viterbi School of Engineering(南加州大学维特比工程学院)

Zhisheng Tang, Mayank Kejriwal

摘要:

近年来,基于transformer的语言表示模型(LRM)在解决自然语言理解中的难题,如问答和文本摘要等方面取得了很好的效果。随着这些模型被集成到现实世界的应用中,评估它们做出理性决策的能力是一个重要的研究议程,具有实际的影响。本文通过一系列精心设计的决策基准和实验,研究了LRM的理性决策能力。受认知科学中经典工作的启发,我们将决策问题建模为赌注。然后,我们调查LRM的能力,选择具有最佳的,或在最低限度,积极的预期收益的结果。通过对四个已建立的LRM进行的大量实验,我们表明,如果首先对具有相同结构的投注问题进行微调,则模型只能“在投注中思考”。修改投注问题的结构,同时仍然保留其基本特征,平均而言,LRM的性能降低了25%以上,尽管绝对性能仍然远高于随机性能。LRM也被发现是更合理的选择结果与非负的预期收益,而不是最佳或严格的正预期收益。我们的研究结果表明,LRM可能适用于依赖认知决策技能的任务,但在他们能够稳健地做出理性决策之前,还需要进行更多的研究。

英文摘要:

In recent years, transformer-based language representation models (LRMs) have achieved state-of-the-art results on difficult natural language understanding problems, such as question answering and text summarization. As these models are integrated into real-world applications, evaluating their ability to make rational decisions is an important research agenda, with practical ramifications. This article investigates LRMs' rational decision-making ability through a carefully designed set of decision-making benchmarks and experiments. Inspired by classic work in cognitive science, we model the decision-making problem as a bet. We then investigate an LRM's ability to choose outcomes that have optimal, or at minimum, positive expected gain. Through a robust body of experiments on four established LRMs, we show that a model is only able to `think in bets' if it is first fine-tuned on bet questions with an identical structure. Modifying the bet question's structure, while still retaining its fundamental characteristics, decreases an LRM's performance by more than 25\%, on average, although absolute performance remains well above random. LRMs are also found to be more rational when selecting outcomes with non-negative expected gain, rather than optimal or strictly positive expected gain. Our results suggest that LRMs could potentially be applied to tasks that rely on cognitive decision-making skills, but that more research is necessary before they can robustly make rational decisions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑