FC-MIR：一种基于帧压缩多模态轨迹推理的移动屏幕感知框架，用于意图感知推荐
FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoning
AI总结：
FC-MIR框架通过帧压缩多模态轨迹推理，提升移动设备上意图感知推荐的效率与实用性，支持轻量级部署并探索生成操作与建议。
摘要：
从移动UI操作轨迹中识别用户意图对于推进UI理解和启用任务自动化代理至关重要。尽管多模态大语言模型（MLLMs）在视频理解任务中表现优异，但其在移动设备上的实时部署受到计算成本高和处理冗余帧效率低的限制。为了解决这些问题，我们提出了FC-MIR框架：通过关键帧采样和自适应拼接，它削减了视觉冗余以提高推理效率，同时整合了最先进的闭源MLLMs或微调模型（例如Qwen3-VL）进行轨迹总结和意图预测。我们进一步扩展任务范围，探索生成预测后操作和搜索建议，并引入一个细粒度指标来评估总结、预测和建议的实用价值。为了严格评估，我们构建了一个涵盖UI-Agents（Agent-I）和真实用户交互（Person-I）场景的UI轨迹数据集。实验结果表明，我们的压缩方法在50%-60%的压缩率下仍能保持性能；闭源和微调的MLLMs在意图总结方面表现强劲，支持潜在的轻量级设备部署。然而，MLLMs仍然在生成有用且“令人惊讶”的建议方面存在困难，仍有改进空间。最后，我们将该框架部署到真实环境中，整合UI感知和UI-Agent代理，为该领域未来的发展奠定基础。
英文摘要：
Identifying user intent from mobile UI operation trajectories is critical for advancing UI understanding and enabling task automation agents. While Multimodal Large Language Models (MLLMs) excel at video understanding tasks, their real-time mobile deployment is constrained by heavy computational costs and inefficient redundant frame processing. To address these issues, we propose the FC-MIR framework: leveraging keyframe sampling and adaptive concatenation, it cuts visual redundancy to boost inference efficiency, while integrating state-of-the-art closed-source MLLMs or fine-tuned models (e.g., Qwen3-VL) for trajectory summarization and intent prediction. We further expand task scope to explore generating post-prediction operations and search suggestions, and introduce a fine-grained metric to evaluate the practical utility of summaries, predictions, and suggestions. For rigorous assessment, we construct a UI trajectory dataset covering scenarios from UI-Agents (Agent-I) and real user interactions (Person-I). Experimental results show our compression method retains performance at 50%-60% compression rates; both closed-source and fine-tuned MLLMs demonstrate strong intent summarization, supporting potential lightweight on-device deployment. However, MLLMs still struggle with useful and "surprising" suggestions, leaving room for improvement. Finally, we deploy the framework in a real-world setting, integrating UI perception and UI-Agent proxies to lay a foundation for future progress in this field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
