The growth of deep learning (DL) relies heavily on huge amounts of labelled data for tasks such as natural language processing and computer vision. Specifically, in image-to-text or image-to-image pipelines, opinion (sentiment) may be inadvertently learned by a model from human-generated image captions. Additionally, learning may be affected by the variety and diversity of the provided captions. While labelling large datasets has largely relied on crowd-sourcing or data-worker pools, evaluating the quality of such training data is crucial.
This study proposes an evaluation method focused on sentiment and semantic richness. That method was applied to the COCO-MS dataset, comprising approximately 150K images with segmented objects and corresponding crowd-sourced captions. We employed pre-trained models (Twitter-RoBERTa-base and BERT-base) to extract sentiment scores and variability of semantic embeddings from captions. The relation of the sentiment score and semantic variability with object categories was examined using multiple linear regression. Results indicate that while most captions were neutral, about 6% of the captions exhibited strong sentiment influenced by specific object categories. Semantic variability of within-image captions remained low and uncorrelated with object categories. Model-generated captions showed less than 1.5% of strong sentiment which was not influenced by object categories and did not correlate with the sentiment of the respective human-generated captions. This research demonstrates an approach to assess the quality of crowd- or worker-sourced captions informed by image content.
CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；LLM Team, Shopee Pte. Ltd.（Shopee私人有限公司大语言模型团队）
MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Image
College of Computer, National University of Defense Technology（国防科技大学计算机学院）；School of Computer and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；CCNU（中国地质大学）
From Visual Feedback to Textual Reviews: A Multi-Agent Vision-Language Framework for Image-Grounded Review Assistance
Indian Institute of Technology Patna（印度理工学院巴特那分校）；Indian Institute of Technology Indore（印度理工学院印多尔分校）；CSIRO（澳大利亚联邦科学与工业研究组织）；University of Western Australia（西澳大学）
GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection
State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)（人工智能安全国家重点实验室，计算技术研究所（ICT），中国科学院（CAS））；University of Chinese Academy of Sciences, CAS（中国科学院大学）；Pengcheng Laboratory（鹏城实验室）；Shenzhen Stock Exchange（深圳证券交易所）
