As general purpose vision models get increasingly effective at a wide set of tasks, it is imperative that they be consistent across the tasks they support. Inconsistent AI models are considered brittle and untrustworthy by human users and are more challenging to incorporate into larger systems that take dependencies on their outputs. Measuring consistency between very heterogeneous tasks that might include outputs in different modalities is challenging since it is difficult to determine if the predictions are consistent with one another. As a solution, we introduce a benchmark dataset, CocoCon, where we create contrast sets by modifying test instances for multiple tasks in small but semantically meaningful ways to change the gold label and outline metrics for measuring if a model is consistent by ranking the original and perturbed instances across tasks. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks, especially for more heterogeneous tasks. To alleviate this issue, we propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets, that improves the multi-task consistency of large unified models while retaining their original accuracy on downstream tasks.
Addressing Imbalance in Multi-Label Data via Label-Specific Distance-based Oversampling
Key Laboratory of Data Engineering and Visual Computing, Chongqing University of Posts and Telecommunications, China(数据工程与视觉计算重点实验室,重庆邮电大学,中国);School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, China(计算机科学与技术学院,重庆邮电大学,中国);State Key Laboratory of Novel Software Technology, Nanjing University, China(新型软件技术国家重点实验室,南京大学,中国);School of Informatics, Aristotle University of Thessaloniki, Greece(信息学院,希腊阿尔蒂米斯大学)
Towards Emotionally Consistent Text-Based Speech Editing: Introducing EmoCorrector and The ECD-TSE Dataset
Johns Hopkins University(约翰霍普金斯大学);Carnegie Mellon University(卡内基梅隆大学);The Chinese University of Hong Kong(香港中文大学);National University of Singapore(新加坡国立大学);Inner Mongolia University(内蒙古大学)
Towards Stable Cross-Domain Depression Recognition under Missing Modalities
School of Future Technology, South China University of Technology(未来技术学院,华南理工大学);PengCheng Laboratory(鹏城实验室);School of Software Engineering, South China University of Technology(软件工程学院,华南理工大学);Artificial Intelligence Research Institute, Shenzhen MSU-BIT University(人工智能研究院,深圳MSU-BIT大学);Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing(粤港澳大湾区情感智能与泛在计算联合实验室);School of Computer Science and Technology, Guangdong University of Technology(计算机科学与技术学院,广东工业大学);Department of Computer Science, City University of Hong Kong(计算机科学系,香港城市大学)