Text-based image retrieval has seen considerable progress in recent years. However, the performance of existing methods suffers in real life since the user is likely to provide an incomplete description of an image, which often leads to results filled with false positives that fit the incomplete description. In this work, we introduce the partial-query problem and extensively analyze its influence on text-based image retrieval. Previous interactive methods tackle the problem by passively receiving users' feedback to supplement the incomplete query iteratively, which is time-consuming and requires heavy user effort. Instead, we propose a novel retrieval framework that conducts the interactive process in an Ask-and-Confirm fashion, where AI actively searches for discriminative details missing in the current query, and users only need to confirm AI's proposal. Specifically, we propose an object-based interaction to make the interactive retrieval more user-friendly and present a reinforcement-learning-based policy to search for discriminative objects. Furthermore, since fully-supervised training is often infeasible due to the difficulty of obtaining human-machine dialog data, we present a weakly-supervised training strategy that needs no human-annotated dialogs other than a text-image dataset. Experiments show that our framework significantly improves the performance of text-based image retrieval. Code is avaiable at https://github.com/CuthbertCai/Ask-Confirm.
Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content
Tianjin University（天津大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring
University of Science and Technology of China（中国科学技术大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Hefei University of Technology（合肥工业大学）
SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent Yuanbao（腾讯元宝）；Tsinghua University（清华大学）；ARC Lab, Tencent（腾讯ARC实验室）；The University of Hong Kong（香港大学）
VAP-Diffusion: Enriching Descriptions with MLLMs for Enhanced Medical Image Generation
College of Biomedical Engineering, Fudan University, Shanghai 200433, China（复旦大学生物医学工程学院）；Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention of Shanghai, Shanghai 200032, China（上海医学影像计算与计算机辅助干预重点实验室）
Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；ByteDance Seed（字节跳动种子）；The Chinese University of Hong Kong（香港中文大学）；City University of Hong Kong（香港城市大学）；BIAI-ZJUT；Zhejiang University of Technology（浙江工业大学）
Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Exploration
Department of Computer Science, University of Warwick（沃里克大学计算机科学系）；Department of Informatics, King’s College London（伦敦国王学院信息学系）；School of Computing, Macquarie University（麦考瑞大学计算机科学学院）；Alan Turing Institute, UK（英国艾伦·图灵研究所）
Cross-Modal Binary Attention: An Energy-Efficient Fusion Framework for Audio-Visual Learning
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School（图林根工业大学和汉诺威医学院医学信息学研究所）；Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)（下萨克森人工智能与因果医学方法中心）
CFMW: Cross-modality Fusion Mamba for Robust Object Detection under Adverse Weather
School of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；USC Viterbi School of Engineering, the University of Southern California（南加州大学维特比工程学院）；College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；School of Robotics, Hunan University（湖南大学机器人学院）
Ask Your Neurons: A Deep Learning Approach to Visual Question Answering
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；UC Berkeley（加州大学伯克利分校）；ICSI（国际计算机科学研究所）
我们解决了一个问答任务，对现实世界的图像，是作为一个视觉图灵测试。通过结合图像表示和自然语言处理的最新进展，我们提出了Ask Your Neurons，一个可扩展的，联合训练的，端到端的公式来解决这个问题。与以前的努力相比，我们正面临着一个多模态的问题，其中的语言输出（答案）的视觉和自然语言输入（图像和问题）的条件。我们提供了额外的见解，通过分析有多少信息只包含在语言部分，我们提供了一个新的人类基线的问题。为了研究人类的共识，这是与在这个具有挑战性的任务中固有的模糊性，我们提出了两个新的指标，并收集额外的答案，扩展原来的DAQUAR数据集DAQUAR-Consensus。此外，我们还将我们的分析扩展到VQA，一个关于图像数据集的大规模问答，在那里我们研究了一些特定的设计选择，并显示了更强大的视觉模型的重要性。与此同时，我们实现了我们的模型的强大性能，仍然使用全局图像表示。最后，基于这样的分析，我们在DAQUAR上改进了我们的Ask Your Neurons，这也导致了这项具有挑战性的任务的更好性能。
Backbone-Conditional Behavior of Modality Gating in Multi-Modal Prostate MRI Segmentation: A 5-Fold Cross-Validation and Gate Mechanism Analysis
The Second Xiangya Hospital of Central South University（中南大学湘雅医学院第二医院）；The Third Xiangya Hospital of Central South University（中南大学湘雅医学院第三医院）；School of Life Sciences, Central South University（中南大学生命科学学院）；Hunan Provincial Key Laboratory of Medical Information Research (Central South University)（湖南省医学信息研究重点实验室（中南大学））；Hunan Provincial Clinical Medical Research Center for Cardiovascular Intelligent Medicine（湖南省心血管智能医学临床医学研究中心）
