Text-to-video (T2V) generation has been recently enabled by transformer-based diffusion models, but current T2V models lack capabilities in adhering to the real-world common knowledge and physical rules, due to their limited understanding of physical realism and deficiency in temporal modeling. Existing solutions are either data-driven or require extra model inputs, but cannot be generalizable to out-of-distribution domains. In this paper, we present PhyT2V, a new data-independent T2V technique that expands the current T2V model's capability of video generation to out-of-distribution domains, by enabling chain-of-thought and step-back reasoning in T2V prompting. Our experiments show that PhyT2V improves existing T2V models' adherence to real-world physical rules by 2.3x, and achieves 35% improvement compared to T2V prompt enhancers. The source codes are available at: https://github.com/pittisl/PhyT2V.
PRISM: Prompt Refinement via Image-grounded Self-rewarding Mechanism for Text-to-Image Generation
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Sun Yat-sen University(中山大学);South China University of Technology(华南理工大学);Guangdong University of Technology(广东工业大学)
Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院);Pengcheng Laboratory(鹏城实验室);School of Computer Science and Technology, Shandong Jianzhu University(山东建筑大学计算机科学与技术学院)
Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement
Department of Computer Science and Engineering, Dongguk University(计算机科学与工程系,东国大学);Department of Computer Science and Artificial Intelligence, Dongguk University(计算机科学与人工智能系,东国大学)
Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement
Department of Computer Science and Information Engineering, National Taiwan University(国家台湾大学计算机科学与信息工程系);National Taiwan University AI Center of Research Excellence(国家台湾大学人工智能研究中心)
Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement
School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
VLM-Guided Iterative Refinement for Surgical Image Segmentation with Foundation Models
School of Automation Science and Engineering, South China University of Technology, Guangzhou, China(自动化科学与工程学院,华南理工大学,广州,中国);Vanderbilt University, Nashville, TN, USA(范德比尔特大学,纳什维尔,田纳西州,美国);The Pennsylvania State University, University Park, PA , USA(宾夕法尼亚州立大学,大学园,宾夕法尼亚州,美国);Peking University, Beijing , China(北京大学,北京,中国);Virginia Commonwealth University, Richmond, VA, USA(弗吉尼亚共同wealth大学,里士满,弗吉尼亚州,美国);University of California San Diego, San Diego, CA, USA(加州大学圣地亚哥分校,圣地亚哥,加利福尼亚州,美国)
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline
Buzzni AI Lab(Buzzni人工智能实验室);KAIST(韩国科学技术院);Mathpresso Inc.(Mathpresso公司);UC Berkeley(加州大学伯克利分校);Fainders AI(Fainders人工智能公司);Seoul National University(首尔大学)
Field Converter: Geometry-Initialized Temporal Residual Refinement for World-Grounded Player Pose Estimation from Soccer Broadcasts
Institut de Biomécanique Humaine Georges Charpak (IBHGC), Arts et Métiers ParisTech(乔治·沙帕克人体生物力学研究所(IBHGC),巴黎高科国立高等工艺学校);French Football Federation (FFF)(法国足球联合会(FFF))
IMPROVE: Iterative Model Pipeline Refinement and Optimization Leveraging LLM Experts
University of Toronto(多伦多大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);New York University(纽约大学)
IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Kuaishou Technology(快手科技);Shenzhen Institute of Advanced Integration Technology, Shenzhen(深圳先进集成技术研究院)