arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.05241cs.CLcs.AIcs.CYcs.ETcs.GT
收藏

大模型战略思维,小模型效率:大语言模型中的心智理论迁移

Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models

Northeastern University(东北大学)

Nunzio Lore, Sepehr Ilami, Babak Heydari

AI总结:

本研究通过将大语言模型在20种社会困境博弈场景中的答案与决策动机用于同系列小模型的问答微调,使小模型对齐大模型心智理论决策能力的表现平均提升46%,且在样本外场景仍有显著提升,为低成本部署高性能战略决策专用小模型提供了可行路径。

摘要:

随着规模更大、更新的大语言模型在战略性心智理论(Theory of Mind, ToM)任务上的表现持续提升,对这些前沿模型的需求也相应增长。然而,它们的部署在算力和时间方面成本都很高。本文中,我们探究了通过微调创建高性能小型专用算法的可行性。为此,我们首先向一个大型预训练模型呈现20个独特场景——这些场景将不同社会情境与各类社会困境博弈相结合,记录其答案,并将这些答案用于同系列更小模型的问答微调。我们的研究聚焦于上下文内的博弈论决策,这正是人类互动发生的领域,既需要心智理论(或其类似表现),也需要对社会动态的理解。因此,小型模型的训练不仅基于大模型给出的答案,还基于大模型提供的决策动机,这些动机应包含应对战略困境和社会线索的建议与指导。我们发现,经过微调的小型语言模型持续缩小了其预训练版本与更大规模同类模型之间的性能差距,且其性能提升延伸到了训练示例之外的领域和情境中,包括包含完全不同博弈结构的样本外场景。以与大模型行为的对齐度(100%代表行为无法区分)为衡量标准,在所有博弈中,小型模型经微调后平均提升了46%。当面对样本外的社会情境和博弈时,微调后的模型仍表现出显著的对齐水平,分别实现了18%和28%的提升。

英文摘要:

As the performance of larger, newer Large Language Models continues to improve for strategic Theory of Mind (ToM) tasks, the demand for these state-of-the-art models increases commensurately. However, their deployment is costly both in terms of processing power and time. In this paper, we investigate the feasibility of creating smaller, highly-performing specialized algorithms by way of fine-tuning. To do this, we first present a large pre-trained model with 20 unique scenarios that combine different social contexts with games of varying social dilemmas, record its answers, and use them for Q&A fine-tuning on a smaller model of the same family. Our focus is on in-context game-theoretic decision-making, the same domain within which human interaction occurs and that requires both a theory of mind (or a semblance thereof) and an understanding of social dynamics. The smaller model is therefore trained not just on the answers provided, but also on the motivations provided by the larger model, which should contain advice and guidelines to navigate both strategic dilemmas and social cues. We find that the fine-tuned smaller language model consistently bridged the gap in performance between the smaller pre-trained version of the model and its larger relative and that its improvements extended in areas and contexts beyond the ones provided in the training examples, including on out-of-sample scenarios that include completely different game structures. On average for all games, through fine-tuning, the smaller model showed a 46% improvement measured as alignment towards the behavior of the larger model, with 100% representing indistinguishable behavior. When presented with out-of-sample social contexts and games, the fine-tuned model still displays remarkable levels of alignment, reaching an improvement of 18% and 28% respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑