Conventional multi-label recognition methods often focus on label confidence, frequently overlooking the pivotal role of partial order relations consistent with human preference. To resolve these issues, we introduce a novel method for multimodal label relevance ranking, named Label Relevance Ranking with Proximal Policy Optimization (LR\textsuperscript{2}PPO), which effectively discerns partial order relations among labels. LR\textsuperscript{2}PPO first utilizes partial order pairs in the target domain to train a reward model, which aims to capture human preference intrinsic to the specific scenario. Furthermore, we meticulously design state representation and a policy loss tailored for ranking tasks, enabling LR\textsuperscript{2}PPO to boost the performance of label relevance ranking model and largely reduce the requirement of partial order annotation for transferring to new scenes. To assist in the evaluation of our approach and similar methods, we further propose a novel benchmark dataset, LRMovieNet, featuring multimodal labels and their corresponding partial order data. Extensive experiments demonstrate that our LR\textsuperscript{2}PPO algorithm achieves state-of-the-art performance, proving its effectiveness in addressing the multimodal label relevance ranking problem. Codes and the proposed LRMovieNet dataset are publicly available at \url{https://github.com/ChazzyGordon/LR2PPO}.
UniNote: A Unified Embedding Model for Multimodal Representation and Ranking
Xiaohongshu Beijing China(小红书北京中国);Shanghai Jiao Tong University(上海交通大学);Huazhong University of Science and Technology(华中科技大学);Beijing Institute of Technology(北京理工大学)
KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs
University of Bari Aldo Moro(巴里大学阿尔多·莫罗分校);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);ZBW - Leibniz Information Centre for Economics(ZBW - 莱布尼茨经济信息中心);University of Kiel(基尔大学)
Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis
Harvard University(哈佛大学);University of California, Santa Cruz(加州大学圣克鲁兹分校);Brown University(布朗大学);Johns Hopkins University(约翰斯·霍普金斯大学);Georgia Institute of Technology(佐治亚理工学院)