Recent advancements in video generation models have significantly improved their ability to follow text prompts. However, the customization of dynamic visual effects, defined as temporally evolving and appearance-driven visual phenomena like object crushing or explosion, remains underexplored. Prior works on motion customization or control mainly focus on low-level motions of the subject or camera, which can be guided using explicit control signals such as motion trajectories. In contrast, dynamic visual effects involve higher-level semantics that are more naturally suited for control via text prompts. However, it is hard and time-consuming for humans to craft a single prompt that accurately specifies these effects, as they require complex temporal reasoning and iterative refinement over time. To address this challenge, we propose P-Flow, a novel training-free framework for customizing dynamic visual effects in video generation without modifying the underlying model. By leveraging the semantic and temporal reasoning capabilities of vision-language models, P-Flow performs test-time prompt optimization, refining prompts based on the discrepancy between the visual effects of the reference video and the generated output. Through iterative refinement, the prompts evolve to better induce the desired dynamic effect in novel scenes. Experiments demonstrate that P-Flow achieves high-fidelity and diverse visual effect customization and outperforms other models on both text-to-video and image-to-video generation tasks. Code is available at https://github.com/showlab/P-Flow.
Visual Prompting in Multimodal Large Language Models: A Survey
UC San Diego(加州大学圣地亚哥分校);Dartmouth College(达特茅斯学院);Rutgers University(罗格斯大学);UC Los Angeles(加州大学洛杉矶分校);Adobe Research(奥多比研究院);The University of New South Wales(新南威尔士大学);CSIRO’s Data61(联邦科学与工业研究组织Data61)
EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensing
Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology(北京理工大学多学科先进研究院);School of Mechatronical Engineering, Beijing Institute of Technology(北京理工大学机电工程学院);Yangtze Delta Region Academy of Beijing Institute of Technology(北京理工大学长三角研究院)
End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting
Beijing Institute of Technology(北京理工大学);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Shenzhen MSU-BIT University(深圳北理莫斯科大学);University of Rochester(罗切斯特大学)
EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Prompting
School of Interdisciplinary Science, Beijing Institute of Technology(交叉科学学院,北京理工大学);College of Computing and Data Science, Nanyang Technological University(计算与数据科学学院,南洋理工大学);National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing Institute of Technology(空间智能信息处理国家重点实验室,北京理工大学);School of Optics and Photonics, Beijing Institute of Technology(光学与 photonics 学院,北京理工大学);State Key Laboratory of Explosion Science and Safety Protection, Beijing(爆炸科学与安全防护国家重点实验室,北京)
VCP-CLIP: A visual context prompting model for zero-shot anomaly segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);CASI Vision Technology CO., LTD.(中智视觉科技有限公司);University of Technology Sydney(悉尼科技大学);Hangzhou Dianzi University(杭州电子科技大学);Tsinghua University(清华大学)
IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting
MAIS, Institute of Automation(自动化研究所信息处理中心);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Research Center of Aerospace Information, Institute of Automation(自动化研究所航空信息研究中心)
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
School of Engineering, Westlake University(西湖大学工程学院);City University of Hong Kong(香港城市大学);Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心);Johns Hopkins University(约翰斯·霍普金斯大学)
Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models
University of Macau(澳门大学);Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Peking University(北京大学);Independent Researcher(独立研究员);Institute of Science Tokyo(东京科学研究院);Morgan Stanley(摩根大通);Halmstad University(哈马碧大学)
Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs
ICG, TU Graz(格拉茨工业大学ICG研究所);CDL-EML(CDL-EML中心);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);JKU(约翰内斯开普勒大学);IBM Research(IBM研究院);Weizmann Institute of Science(魏茨曼科学研究所);University of Bonn(波恩大学)
Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
ASUS Intelligent Cloud Services (AICS)(华硕智能云服务(AICS));ASUS Global Pte. Ltd.(华硕全球私人有限公司);Centre for Frontier AI Research (CFAR)(前沿人工智能研究中心(CFAR));Institute of High Performance Computing (IHPC)(高性能计算研究所(IHPC));Agency for Science, Technology and Research (A*STAR)(新加坡科学、技术与研究局(A*STAR))
LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation
Rutgers University(罗格斯大学);Zhejiang University(浙江大学);Red Hat AI Innovation(红帽人工智能创新中心);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);NVIDIA Research(英伟达研究院)
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));vivo Mobile Communication Co., Ltd.(维沃移动通信有限公司);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Pengcheng Laboratory(鹏城实验室)
University of Science and Technology of China(中国科学技术大学);Microsoft Cloud AI(微软云人工智能部门);Wormpex AI Research(Wormpex人工智能研究院)
提出了Diversity-Aware Meta Visual Proximating ~(DAM-VP),它是一种高效的提示方法,用于将预先训练好的模型转移到具有冻结主干的下游任务中。视觉提示中的一个具有挑战性的问题是,图像数据集有时具有很大的数据多样性,而每个数据集的通用提示很难处理复杂的分布偏移向原始预训练数据分布正确。为了解决这个问题,我们提出了一个数据集的多样性意识的提示策略,其初始化是由一个元提示。具体来说,我们以多样性自适应的方式将下游数据集聚类成小的同质子集,每个子集都有自己的提示单独优化。这种分而治之的设计大大降低了优化的难度,并显着提高了提示性能。此外,所有提示都使用元提示进行初始化,该元提示是在多个数据集上学习的。这是一个自举的范例,关键的观察是,从以前的数据集学习的提示知识可以帮助提示更快地收敛,并在新的数据集上表现得更好。在推理过程中,我们根据输入和每个子集之间的特征距离为每个输入动态选择适当的提示。通过大量的实验,我们的DAM-VP表现出了卓越的效率和有效性,在不同预训练模型的一系列下游数据集中明显超过了以前的提示方法。我们的代码可从以下网址获得:\url{https://github.com/shikiw/DAM-VP}。