Action Reimagined: 面向动态人类动作的文本转姿态视频编辑
Action Reimagined: Text-to-Pose Video Editing for Dynamic Human Actions
AI总结：
针对现有视频编辑难以修改人类动作的问题，提出ReimaginedAct方法，结合LLM、Grounded-SAM和姿态检索实现文本驱动的动态动作编辑，还构建了WhatifVideo-1.0数据集，验证了方法的有效性。
摘要：
我们提出了一种新颖的文本转姿态视频编辑方法ReimaginedAct。现有视频编辑任务仅局限于属性、背景和风格的改动，我们的方法旨在预测视频中开放式的人类动作变化。此外，该方法不仅能接收直接的指令文本提示，还能处理“假设”类问题以预测可能的动作变化。ReimaginedAct包含视频理解、推理和编辑模块。首先，利用大语言模型（LLM）生成指令或问题的合理答案，该答案随后用于两个用途：一是提示Grounded-SAM生成相关人物的边界框，二是检索我们收集的一组姿态视频，用于编辑人类动作。接着，将检索到的姿态视频与检测到的人物结合，修改从原始视频中提取的姿态。我们还采用了时间步混合模块，确保编辑后的视频除必要修改区域外，保留原有内容。为推动文本转姿态视频编辑领域的研究，我们推出了新的评估数据集WhatifVideo-1.0。该数据集包含不同场景、不同难度级别的视频，以及对应的问题和文本提示。实验结果表明，现有视频编辑方法难以胜任人类动作编辑任务，而我们的方法能够实现有效的动作编辑，甚至可基于反事实问题完成想象式编辑。
英文摘要：
We introduce a novel text-to-pose video editing method, ReimaginedAct. While existing video editing tasks are limited to changes in attributes, backgrounds, and styles, our method aims to predict open-ended human action changes in video. Moreover, our method can accept not only direct instructional text prompts but also `what if' questions to predict possible action changes. ReimaginedAct comprises video understanding, reasoning, and editing modules. First, an LLM is utilized initially to obtain a plausible answer for the instruction or question, which is then used for (1) prompting Grounded-SAM to produce bounding boxes of relevant individuals and (2) retrieving a set of pose videos that we have collected for editing human actions. The retrieved pose videos and the detected individuals are then utilized to alter the poses extracted from the original video. We also employ a timestep blending module to ensure the edited video retains its original content except where necessary modifications are needed. To facilitate research in text-to-pose video editing, we introduce a new evaluation dataset, WhatifVideo-1.0. This dataset includes videos of different scenarios spanning a range of difficulty levels, along with questions and text prompts. Experimental results demonstrate that existing video editing methods struggle with human action editing, while our approach can achieve effective action editing and even imaginary editing from counterfactual questions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
