Face recognition (FR) systems are widely deployed in critical applications, making their reliability and robustness across diverse populations and conditions essential. Standard evaluation of FR systems typically relies on datasets such as LFW to estimate average recognition accuracy. Some benchmarks also capture coarse-grained intra-identity variations such as aging, pose, and lighting. However, human faces undergo more fine-grained changes, including appearance changes such as hairstyles and makeup, that are underrepresented in existing benchmarks. Counterfactual evaluation provides a method to assess FR robustness under such fine-grained variations. Existing counterfactual face datasets synthesized with image generators, however, are limited in attribute coverage due to the use of humans for verification in the pipeline.
We propose CounterFace, a new counterfactual evaluation dataset comprising 20 facial attributes and 8 demographic factors, exceeding prior synthetic face datasets by 14 attributes and 2 demographics. The dataset is generated using a fully automated pipeline based on off-the-shelf image generators with custom verifiers, removing human need for verification. CounterFace contains 11,821 counterfactual face pairs, and a post-hoc user study confirms the faithfulness of the generated counterfactuals. We evaluate two commercial and four open-source FR systems (AWS Rekognition, Face++, AdaFace, MagFace, ArcFace, FaceNet) across 160 attribute-demographic combinations. Our dataset helps in the isolation of precise failure modes for individual systems unlike standard evaluation benchmarks. Results indicate that the performance degradation varies across attributes and demographics for all six systems and occluding attributes (e.g., facemask and facial hair) universally degrade performance.
Attention mechanism has demonstrated great potential in fine-grained visual recognition tasks. In this paper, we present a counterfactual attention learning method to learn more effective attention based on causal inference. Unlike most existing methods that learn visual attention based on conventional likelihood, we propose to learn the attention with counterfactual causality, which provides a tool to measure the attention quality and a powerful supervisory signal to guide the learning process. Specifically, we analyze the effect of the learned visual attention on network prediction through c…
GCAM: Gaussian and causal-attention model of food fine-grained recognition
School of Computer and Information, Hefei University of Technology（合肥工业大学计算机与信息学院）；School of Food Science and Engineering, Northern University for Nationalities（北方民族大学食品科学与工程学院）；School of Electrical Engineering, Xinjiang University（新疆大学电气工程学院）；School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）
Boston University（波士顿大学）；Georgetown University（乔治城大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Modern deep neural networks tend to be evaluated on static test sets. One shortcoming of this is the fact that these deep neural networks cannot be easily evaluated for robustness issues with respect to specific scene variations. For example, it is hard to study the robustness of these networks to variations of object scale, object pose, scene lighting and 3D occlusions. The main reason is that collecting real datasets with fine-grained naturalistic variations of sufficient scale can be extremely time-consuming and expensive. In this work, we present Counterfactual Simulation Testing, a counte…
National University of Singapore（新加坡国立大学）；Université de Montréal（蒙特利尔大学）；Zhejiang University（浙江大学）
Content-Based Image Retrieval (CIR) aims to search for a target image by concurrently comprehending the composition of an example image and a complementary text, which potentially impacts a wide variety of real-world applications, such as internet search and fashion retrieval. In this scenario, the input image serves as an intuitive context and background for the search, while the corresponding language expressly requests new traits on how specific characteristics of the query image should be modified in order to get the intended target image. This task is challenging since it necessitates lea…
FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation
Zhejiang University（浙江大学）；Hokkaido University（北海道大学）；The University of Queensland（昆士兰大学）；Shanghai AI Laboratory（上海人工智能实验室）；Dalian University of Technology（大连理工大学）
CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale
The Key Laboratory of Smart Human-Computer Interaction（智能人机交互关键实验室）；Wearable Technology of Shaanxi Province, Xidian University, Xi'an, China（陕西 wearable 技术, 西安电子科技大学, 中国）；Nanjing University of Science and Technology（南京理工大学）
CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Sun Yat-sen University-Shenzhen（中山大学（深圳））
Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detection
Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）；Xi’an Jiaotong University（西安交通大学）；Australian National University（澳大利亚国立大学）；Tsinghua University（清华大学）
WBCAtt+: Fine-Grained Pixel-Level Morphological Annotations for White Blood Cell Images
Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technological University（快速丰富目标搜索（ROSE）实验室，电气与电子工程学院，南洋理工大学）；Shanghai University of Finance and Economics（上海财经大学）
Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation
South China University of Technology（华南理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Northwestern Polytechnical University（西北工业大学）；The Hong Kong Polytechnic University（香港理工大学）
