发表机构
University of Trento; Technovative Solutions Ltd; Massachusetts Institute of Technology; Free University of Bozen Bolzano; University of Edinburgh（特伦托大学; 泰诺创新解决方案有限公司; 麻省理工学院; 博岑-波尔扎诺自由大学; 爱丁堡大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文探究不同对话游戏间的知识迁移，通过微调LLM模型开展两项分析，发现视觉空间类游戏迁移效果最佳，基于相似性的方法难以捕捉可迁移性模式。
AI 中文摘要
对话游戏是一种具有挑战性的场景，完成任务需要复杂的认知技能，同时还需与其他玩家协调。由于语言既是理解游戏规则的接口，也是执行动作的接口，因此有理由认为，在特定语言游戏上进行训练会提升特定能力，这些能力可能也与其他任务相关。受此原理启发，本文探究知识如何在不同对话游戏间迁移。我们通过在clembench套件（Chalamalasetti等人，2023）中的游戏上微调LLM模型来研究可迁移性，并开展两项分析：i）使用Zamir等人（2018）提出的二元整数优化程序，以任务性能为主要指标推导任务可迁移性图；ii）为每个游戏计算任务向量（Ilharco等人，2022），以研究微调模型间的相似性及其任务可迁移性。在第一项分析中，我们发现部分游戏从迁移中获得的收益多于微调，且视觉空间类游戏（例如探索类游戏）的迁移效果最佳。而在任务向量分析中，我们发现基于相似性的方法能捕捉游戏角色关系，但几乎无法捕捉可迁移性模式，这表明需要更复杂的指标。
英文摘要
Dialogue games represent a challenging setting where complex cognitive skills are required to accomplish tasks while coordinating with other players. Considering that language represents an interface for both understanding the game rules and executing actions, it is reasonable to assume that training on a specific language game will enhance specific capabilities that might be relevant for other tasks as well. Motivated by this rationale, in this paper, we investigate how knowledge transfers across different dialogue games. We study transferability by finetuning LLM models on games from the clembench suite (Chalamalasetti et al., 2023) and performing two analyses: i) we derive a task-transferability graph using a binary integer optimization program from Zamir et al. (2018), using task performance as the main metric; and ii) we compute task vectors (Ilharco et al., 2022) for each game to study similarities across finetuned models and their task transferability. In our first analysis, we find that some games benefit more from transfer than finetuning, and that the visuospatial family (e.g., exploration games) transfers best. With our task vector analysis instead, we find that similarity-based approaches capture game-role relationships but almost no transferability patterns, suggesting that more complex metrics are required.
