Sharingan：从桌面录制中提取用户操作序列
Sharingan: Extract User Action Sequence from Desktop Recordings
AI总结：
本文提出两种基于VLM的方法（DF和DiffF）从桌面录制中提取用户操作序列，实验表明DF方法准确率达70%-80%且操作序列可通过RPA重播，是VLM在该领域的首次应用。
摘要：
用户活动的视频录制，特别是桌面录制，为理解用户行为和自动化流程提供了丰富的数据源。然而，尽管视觉语言模型（VLMs）取得了进展并在视频分析中越来越多地被使用，从桌面录制中提取用户操作仍然是一个未被充分探索的领域。本文通过提出两种基于VLM的全新用户操作提取方法来填补这一空白：直接基于帧的方法（DF），将采样帧直接输入VLMs；以及差分基于帧的方法（DiffF），结合通过计算机视觉技术检测到的显式帧差异。我们使用一个基础的自建数据集和一个改编自先前工作的高级基准来评估这些方法。我们的结果表明，DF方法在识别用户操作方面达到了70%到80%的准确率，提取的操作序列可通过机器人流程自动化重新播放。我们发现，尽管VLMs显示出潜力，但结合显式UI变化可能会降低性能，使得DF方法更加可靠。这项工作代表了VLMs首次应用于从桌面录制中提取用户操作序列，为未来的研究贡献了新方法、基准和见解。
英文摘要：
Video recordings of user activities, particularly desktop recordings, offer a rich source of data for understanding user behaviors and automating processes. However, despite advancements in Vision-Language Models (VLMs) and their increasing use in video analysis, extracting user actions from desktop recordings remains an underexplored area. This paper addresses this gap by proposing two novel VLM-based methods for user action extraction: the Direct Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and the Differential Frame-Based Approach (DiffF), which incorporates explicit frame differences detected via computer vision techniques. We evaluate these methods using a basic self-curated dataset and an advanced benchmark adapted from prior work. Our results show that the DF approach achieves an accuracy of 70% to 80% in identifying user actions, with the extracted action sequences being re-playable though Robotic Process Automation. We find that while VLMs show potential, incorporating explicit UI changes can degrade performance, making the DF approach more reliable. This work represents the first application of VLMs for extracting user action sequences from desktop recordings, contributing new methods, benchmarks, and insights for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
