Understanding the inner representation of a neural network helps users improve models. Concept-based methods have become a popular choice for explaining deep neural networks post-hoc because, unlike most other explainable AI techniques, they can be used to test high-level visual "concepts" that are not directly related to feature attributes. For instance, the concept of "stripes" is important to classify an image as a zebra. Concept-based explanation methods, however, require practitioners to guess and manually collect multiple candidate concept image sets, making the process labor-intensive and prone to overlooking important concepts. Addressing this limitation, in this paper, we frame concept image set creation as an image generation problem. However, since naively using a standard generative model does not result in meaningful concepts, we devise a reinforcement learning-based preference optimization (RLPO) algorithm that fine-tunes a vision-language generative model from approximate textual descriptions of concepts. Through a series of experiments, we demonstrate our method's ability to efficiently and reliably articulate diverse concepts that are otherwise challenging to craft manually.
Discovering Cyclists' Visual Preferences Through Shared Bike Trajectories and Street View Images Using Inverse Reinforcement Learning
Harbin Institute of Technology Shenzhen(哈尔滨工业大学(深圳));Shenzhen Technology University(深圳技术大学);Macau University of Science and Technology(澳门科技大学);Peking University(北京大学)
MVP-CBM:Multi-layer Visual Preference-enhanced Concept Bottleneck Model for Explainable Medical Image Classification
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China(中国科学技术大学生物医学工程学院、生命科学与医学系);Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏等候研院);Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology(江苏省多模态数字孪生技术省级重点实验室);Medical Business Department, iFlytek Co., Ltd(iFlytek有限公司医疗业务部);State Key Laboratory of Precision and Intelligent Chemistry, USTC(精准与智能化学国家重点实验室)
AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(中国教育部多媒体可信感知与高效计算重点实验室,厦门大学);Tencent YouTu Lab(腾讯YouTu实验室)
MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models
School of Computer Science and Engineering, MoE Key Laboratory of Information Technology, Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University, Guangzhou 510006, China(计算机科学与工程学院,信息技术MOE实验室,广东省信息安全技术重点实验室,中山大学,广州510006,中国);State Key Laboratory of Mathematical Engineering and Advanced Computing(数学工程与先进计算国家重点实验室);Department of Computer and Information Science, University of Macau.(计算机与信息科学系,澳门大学)
Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks
Hong Kong University of Science and Technology(香港科技大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Harvard University(哈佛大学);HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute(香港科技大学深港协同创新研究院)
From Images to Insights: Explainable Biodiversity Monitoring with Plain Language Habitat Explanations
Leibniz Centre for Agricultural Landscape Research (ZALF)(莱比锡农业景观研究中心(ZALF));Brandenburg University of Technology Cottbus–Senftenberg(勃兰登堡技术大学科特博斯-森芬根堡)