arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.12937cs.AIcs.CV
收藏

Yanyun-3: 通过视觉-语言模型实现跨平台战略游戏操作

Yanyun-3: Enabling Cross-Platform Strategy Game Operation with Vision-Language Models

Nanjing University of Science and Technology(南京理工大学);Nanjing University(南京大学)

Guoyan Wang, Yanyan Huang, Chunlin Chen, Lifeng Wang, Yuxiang Sun

AI总结:

Yanyun-3通过整合视觉-语言模型和结构化多模态数据组织,实现了跨平台战略游戏操作的自动化,提升了任务执行效率和泛化能力。

摘要:

跨平台战略游戏自动化仍面临挑战,由于用户界面多样性和动态战场环境。现有视觉-语言模型(VLMs)在异构平台上的泛化能力不足,且在界面理解和动作执行的精度上存在不足。我们引入Yanyun-3,一种基于VLM的代理,集成了Qwen2.5-VL进行视觉推理和UI-TARS进行界面执行。我们提出了一种新的数据组织原则——组合粒度,以区分多模态数据的内样本融合和跨样本混合(静态图像、多图像序列和视频)。模型使用QLoRA在三个战略游戏平台上进行微调。最优策略(M*V+S)在BLEU-4得分上比全融合提高了12.98倍,在推理时间上减少了63%。Yanyun-3能够在不同平台上执行核心任务(例如目标选择、资源分配)而无需平台特定的调优。我们的发现表明,结构化的多模态数据组织显著增强了VLM在具身任务中的性能。Yanyun-3为GUI自动化提供了一个可推广的框架,对机器人和自主系统具有更广泛的影响。

英文摘要:

Cross-platform strategy game automation remains a challenge due to diverse user interfaces and dynamic battlefield environments. Existing Vision--Language Models (VLMs) struggle with generalization across heterogeneous platforms and lack precision in interface understanding and action execution. We introduce Yanyun-3, a VLM-based agent that integrates Qwen2.5-VL for visual reasoning and UI-TARS for interface execution. We propose a novel data organization principle -- combination granularity -- to distinguish intra-sample fusion and inter-sample mixing of multimodal data (static images, multi-image sequences, and videos). The model is fine-tuned using QLoRA on a curated dataset across three strategy game platforms. The optimal strategy (M*V+S) achieves a 12.98x improvement in BLEU-4 score and a 63% reduction in inference time compared to full fusion. Yanyun-3 successfully executes core tasks (e.g., target selection, resource allocation) across platforms without platform-specific tuning. Our findings demonstrate that structured multimodal data organization significantly enhances VLM performance in embodied tasks. Yanyun-3 offers a generalizable framework for GUI automation, with broader implications for robotics and autonomous systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑