arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.07516cs.CLcs.AIcs.LG
收藏

通过增强覆盖的潜在动作控制多模态对话代理

Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actions

School of Computer Science, Wuhan University(武汉大学计算机学院);Tongyi Lab(通义实验室);Zhongguancun Academy(中关村学院)

Yongqi Li, Hao Lang, Tieyun Qian, Yongbin Li

AI总结:

本文提出通过增强覆盖的潜在动作空间改进多模态对话代理的强化学习微调,利用跨模态投影器提升鲁棒性,在两个对话任务中优于基线方法。

摘要:

视觉-语言模型日益被用于多模态对话代理(MCAs)以处理多样化的对话任务。最近,强化学习(RL)被广泛探索以适应各种人机交互场景。尽管在泛化性能上有所提升,但通过RL微调MCAs仍面临处理极大量文本标记空间的挑战。为此,我们学习了一个紧凑的潜在动作空间进行RL微调。具体来说,我们采用从观察学习的机制来构建潜在动作空间的代码本,其中未来观察被用来估计当前潜在动作,进而用于重构未来观察。然而,配对图像-文本数据的稀缺性阻碍了学习具有足够覆盖范围的代码本。因此,我们利用配对图像-文本数据和纯文本数据来构建潜在动作空间,使用跨模态投影器将文本嵌入转换为图像-文本嵌入。我们首先在配对图像-文本数据上初始化跨模态投影器,然后在大规模纯文本数据上进一步训练它,使用一种新的循环一致性损失来增强其鲁棒性。我们证明了基于潜在动作的方法在两个对话任务中优于竞争基线,跨各种RL算法。

英文摘要:

Vision-language models are increasingly employed as multimodal conversational agents (MCAs) for diverse conversational tasks. Recently, reinforcement learning (RL) has been widely explored for adapting MCAs to various human-AI interaction scenarios. Despite showing great enhancement in generalization performance, fine-tuning MCAs via RL still faces challenges in handling the extremely large text token space. To address this, we learn a compact latent action space for RL fine-tuning instead. Specifically, we adopt the learning from observation mechanism to construct the codebook for the latent action space, where future observations are leveraged to estimate current latent actions that could further be used to reconstruct future observations. However, the scarcity of paired image-text data hinders learning a codebook with sufficient coverage. Thus, we leverage both paired image-text data and text-only data to construct the latent action space, using a cross-modal projector for transforming text embeddings into image-text embeddings. We initialize the cross-modal projector on paired image-text data, and further train it on massive text-only data with a novel cycle consistency loss to enhance its robustness. We show that our latent action based method outperforms competitive baselines on two conversation tasks across various RL algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑