对于文本到图像生成而言,自动将用户提供的自然语言提示精炼为系统所偏好的关键词丰富型提示,对用户体验至关重要。这一提示精炼过程类似于将提示从“用户语言”翻译为“系统语言”。然而,此类平行语料的稀缺使得训练提示精炼模型变得困难。受零样本机器翻译技术的启发,我们提出了基于图像枢轴的提示精炼方法(Prompt Refinement with Image Pivot,PRIP)。PRIP创新性地利用用户偏好图像的潜在表示作为用户语言与系统语言之间的中间“枢轴”。它将精炼过程分解为两个数据丰富的任务:从用户语言推断用户偏好图像的表示,以及随后将图像表示翻译为系统语言。因此,该方法能够利用丰富的数据进行训练。大量实验表明,PRIP显著优于多种基线方法,并能以零样本方式有效迁移到未见过的系统。
英文摘要:
For text-to-image generation, automatically refining user-provided natural language prompts into the keyword-enriched prompts favored by systems is essential for the user experience. Such a prompt refinement process is analogous to translating the prompt from "user languages" into "system languages". However, the scarcity of such parallel corpora makes it difficult to train a prompt refinement model. Inspired by zero-shot machine translation techniques, we introduce Prompt Refinement with Image Pivot (PRIP). PRIP innovatively uses the latent representation of a user-preferred image as an intermediary "pivot" between the user and system languages. It decomposes the refinement process into two data-rich tasks: inferring representations of user-preferred images from user languages and subsequently translating image representations into system languages. Thus, it can leverage abundant data for training. Extensive experiments show that PRIP substantially outperforms a wide range of baselines and effectively transfers to unseen systems in a zero-shot manner.
Refinement Is Inherently Editable: Training-Free Prompt-to-Prompt Image Editing with Generative Refinement Network
City University of Hong Kong (Dongguan)(香港城市大学(东莞));City University of Hong Kong(香港城市大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
PRISM: Prompt Refinement via Image-grounded Self-rewarding Mechanism for Text-to-Image Generation
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Sun Yat-sen University(中山大学);South China University of Technology(华南理工大学);Guangdong University of Technology(广东工业大学)
Culture-TRIP: Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement
Department of Computer Science and Engineering, Dongguk University(计算机科学与工程系,东国大学);Department of Computer Science and Artificial Intelligence, Dongguk University(计算机科学与人工智能系,东国大学)
TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation
Nara Institute of Science and Technology(奈良科学技術研究所);The University of Tokyo(东京大学);Chungnam National University(Chungnam国立大学);Institute of Science Tokyo(东京科学研究所)
SRPR-Net: Semantic and Relational Prompt Refinement for Automated SAM-based Instance Segmentation
Ruijin-XJTLU Intelligent Medicine Institute, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine(瑞金-西交利物浦智能医学研究所,瑞金医院,上海交通大学医学院);School of Biomedical Engineering, Shanghai Jiao Tong University(上海交通大学生物医学工程学院)
CP3: Unifying Point Cloud Completion by Pretrain-Prompt-Predict Paradigm
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Shanghai AI Laboratory(上海人工智能实验室)
CamoSAM2: SAM2-oriented Prompt Auto-Refinement for Video Camouflaged Object Detection
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University(合成视觉基础科学国家重点实验室,四川大学);College of Computer Science, Sichuan University(计算机学院,四川大学)
Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning
School of Software, Shandong University(山东大学软件学院);AIM Lab, Faculty of Engineering, Monash University(莫纳什大学工程学院AIM实验室);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Department of Electrical and Computer Engineering, University of British Columbia(不列颠哥伦比亚大学电气与计算机工程系);Vector Institute, Canada(加拿大向量研究所)
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
School of Engineering, Westlake University(西湖大学工程学院);City University of Hong Kong(香港城市大学);Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心);Johns Hopkins University(约翰斯·霍普金斯大学)
CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement
Business School, Shandong University of Technology, Shandong, China(山东科技大学商学院);Institute of International Language Services Studies, Macau Millennium College, Macau SAR, China(国际语言服务研究所,澳门 Millennium 学院)
PromptSR: Cascade Prompting for Lightweight Image Super-Resolution
School of Electrical and Electronics Engineering, Nanyang Technological University, Singapore(南洋理工大学电子与电气工程学院);School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China(华中科技大学电子信息与通信学院);Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong(香港理工大学电子与电气工程系)
Robust Box Prompt based SAM for Medical Image Segmentation
National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Medical School, Shenzhen University(深圳大学医学院生物医学工程学院国家区域医学超声关键技术工程实验室);Shenzhen University(深圳大学);Shenzhen RayShape Medical Technology Co., Ltd(深圳睿形医疗科技有限公司);First Affiliated Hospital, Southern University of Science and Technology, Shenzhen People’s Hospital(南方科技大学第一附属医院(深圳市人民医院))