University of Washington(华盛顿大学);Salesforce Research(Salesforce研究院);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校)
Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, Etash Guha, Silvio Savarese, Ludwig S… 展开作者
Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, Etash Guha, Silvio Savarese, Ludwig Schmidt, Yejin Choi, Caiming Xiong, Ran Xu
We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that bridges the gap between descriptive synthetic captions and factual web-scale alt-text. KALE augments synthetic dense image captions with web-scale alt-text to generate factually grounded image captions. Our two-stage approach leverages large vision-language models and language models to create knowledge-augmented captions, which are then used to train a specialized VLM for scaling up the dataset. We train vision-language models on KALE and demonstrate improvements on vision-language tasks. Our experiments show the utility of KALE for training more capable and knowledgeable multimodal models. We release the KALE dataset at https://huggingface.co/datasets/Salesforce/blip3-kale
RACap: Relation-Aware Prompting for Lightweight Retrieval-Augmented Image Captioning
School of biological science and medical engineering, Southeast University, China(生物科学与医学工程学院,东南大学,中国);Department of Neurosurgery, The second affiliated hospital of Xinjiang Medical University, China(神经外科部门,新疆医科大学第二附属医院,中国)
SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering
School of Computer Science, Hubei University, Wuhan, China(湖北大学计算机学院);Hubei Key Laboratory of Big Data Intelligent Analysis and Application (Hubei University)(湖北大数据智能分析与应用重点实验室);Key Laboratory of Intelligent Sensing System and Security (Hubei University)(智能传感系统与安全重点实验室);Institute of Vocational Education, Guangdong Industry Polytechnic University(广东行业职业大学职业教育学院);Shandong Police College(山东警察学院)
TPCap: Unlocking Zero-Shot Image Captioning with Trigger-Augmented and Multi-Modal Purification Modules
Faculty of Information Engineering and Automation, Kunming University of Science and Technology(昆明理工大学信息工程与自动化学院);Yunnan Key Laboratory of Computer Technologies Application(云南省计算机技术应用重点实验室);Hongyun Honghe Group Honghe Cigarette Factory(红云红河集团红河卷烟厂)
SEMT: Static-Expansion-Mesh Transformer Network Architecture for Remote Sensing Image Captioning
Austrian Institute of Technology(奥地利技术研究院);University of Technology of Troyes(图卢兹技术大学);Ho Chi Minh University of Technology(胡志明技术大学);Ton Duc Thang University(-ton Duc Thang大学)
DualCap: Enhancing Lightweight Image Captioning via Dual Retrieval with Similar Scenes Visual Prompts
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
Shanghai Jiao Tong University(上海交通大学);SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University(上海交通大学巴黎卓越工程师学院);The Chinese University of Hong Kong(香港中文大学)
Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Model
School of Cyber Science and Technology, University of Science and Technology of China(信息科学技术学院,中国科学技术大学);China Academy of Electronics and Information Technology(电子信息技术研究院)
Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning
AI Platform OneNexus, OneMount(OneMount AI平台OneNexus);School of Electronic Engineering, Soongsil University(崇实大学电子工程学院);MoAdata(MoAdata公司);University of Economics Ho Chi Minh City (UEH)(胡志明市经济大学)
RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI)
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)
图像描述传统上依赖于基于参考的自动评估,其中将机器字幕与人类书写的字幕进行比较。这与人类评估字幕质量的无参考方式形成对比。在本文中,我们报告了令人惊讶的实证发现,CLIP(Radford et al.,2021),一个跨模态模型预训练的400 M图像+字幕对从网络上,可以用于鲁棒的自动评估图像描述,而不需要参考。跨越几个语料库的实验表明,我们的新的无参考指标,CLIPScore,实现了最高的相关性与人类的判断,优于现有的基于参考的指标,如CIDER和SPICE。信息增益实验表明,CLIPScore,其紧密关注的图像-文本兼容性,是补充现有的基于参考的指标,强调文本-文本的相似性。因此,我们还提出了一个参考增强版本,RefCLIPScore,它实现了更高的相关性。除了文字描述任务之外,几个案例研究揭示了CLIPScore表现良好的领域(剪贴画图像,替代文本评级),但与基于参考的指标相比,它也相对较弱,例如,新闻标题需要更丰富的上下文知识。
Performance Improvement of Language-Queried Audio Source Separation Based on Caption Augmentation From Large Language Models for DCASE Challenge 2024 Task 9
Gwangju Institute of Science and Technology(光州科学技术院);Aunion AI, Co. Ltd.(Aunion AI有限公司)