本文探讨了大型语言模型(LLMs),如GPT-4和LLaMA 2,在零样本设置下的图像分享能力。为了便于对LLMs进行全面评估,我们引入了PhotoChat++数据集,该数据集包含丰富的标注信息(即意图、触发句、图像描述和显著信息)。此外,我们提出了无梯度且可扩展的Decide, Describe, and Retrieve(DribeR)框架。通过大量实验,我们解锁了配备LLMs的DribeR在零样本提示下的图像分享能力,其中ChatGPT取得了最佳性能。我们的发现还揭示了LLMs在零样本条件下涌现出的图像分享能力,验证了DribeR的有效性。我们使用该框架在两个真实场景中展示了其实用性和有效性:(1)人机交互和(2)数据集增强。据我们所知,这是第一项在零样本设置下评估各种LLMs图像分享能力的研究。我们在https://github.com/passing2961/DribeR上公开了源代码和数据集。
英文摘要:
This paper explores the image-sharing capability of Large Language Models (LLMs), such as GPT-4 and LLaMA 2, in a zero-shot setting. To facilitate a comprehensive evaluation of LLMs, we introduce the PhotoChat++ dataset, which includes enriched annotations (i.e., intent, triggering sentence, image description, and salient information). Furthermore, we present the gradient-free and extensible Decide, Describe, and Retrieve (DribeR) framework. With extensive experiments, we unlock the image-sharing capability of DribeR equipped with LLMs in zero-shot prompting, with ChatGPT achieving the best performance. Our findings also reveal the emergent image-sharing ability in LLMs under zero-shot conditions, validating the effectiveness of DribeR. We use this framework to demonstrate its practicality and effectiveness in two real-world scenarios: (1) human-bot interaction and (2) dataset augmentation. To the best of our knowledge, this is the first study to assess the image-sharing ability of various LLMs in a zero-shot setting. We make our source code and dataset publicly available at https://github.com/passing2961/DribeR.
R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO
Nanyang Technological University(南洋理工大学);ByteDance(字节跳动);Tsinghua University(清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);The University of Sydney(悉尼大学)
Computational Lesions in Multilingual Language Models Separate Shared and Language-specific Brain Alignment
The University of Manchester(曼彻斯特大学);City University of Hong Kong(香港城市大学);The Hong Kong Polytechnic University(香港理工大学);Institute of Automation, CAS(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Johns Hopkins University(约翰霍普金斯大学)
Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages
Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE(民族语言智能分析与安全治理重点实验室);Minzu University of China(民族大学);Shanghai Jiao Tong University(上海交通大学)
1-800-SHARED-TASKS @ NLU of Devanagari Script Languages: Detection of Language, Hate Speech, and Targets using LLMs
Pulchowk Campus, IoE(工程学院(IoE)普尔乔克校区);University of Wisconsin(威斯康星大学);NUST(国立科技大学);Cohere For AI Community(Cohere AI社区);University of Maryland(马里兰大学);University of Maryland, College Park(马里兰大学帕克分校)
本文针对 CHiPSAL 2025 共享任务,利用 MuRIL、IndicBERT、Gemma-2 及 focal loss 等技术处理 Devanagari 语言检测、仇恨言论及目标识别,在三个子任务中取得有竞争力的 F1 结果。
Why Pretraining Fails to Share Cross-Lingual Knowledge
Weizmann Institute of Science(魏茨曼科学研究所);Bar-Ilan University(巴伊兰大学);Johns Hopkins University(约翰斯·霍普金斯大学);A*STAR(新加坡科技研究局);University of Washington(华盛顿大学);MIT(麻省理工学院);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室)
Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communication
Leiden Institute of Advanced Computer Science, Leiden University, Netherlands(莱顿高级计算机科学研究所,莱顿大学,荷兰);School of Computing, University of Kent, United Kingdom(计算学院,肯特大学,英国);Cognitive Psychology Unit, Institute of Psychology, Leiden, the Netherlands(认知心理学单元,心理学研究所,莱顿,荷兰)
SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models
National University of Singapore (Chongqing) Research Institute(新加坡国立大学(重庆)研究院);Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles(重庆智能网联车辆可信感知与交互技术重点实验室);National University of Singapore(新加坡国立大学);Hebrew University of Jerusalem(耶路撒冷希伯来大学);State Key Laboratory of Intelligent Vehicle Safety Technology, Chongqing, China(重庆智能车辆安全技术国家重点实验室);CHONGQING CHANGAN AUTOMOBILE Co., Ltd(重庆长安汽车有限公司)
Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View
Google Research(谷歌研究院);Cornell University(康奈尔大学);Deepmind(深度思维)
Touchdown数据集(Chen等人,2019)提供了人类注释者在纽约市街道上导航和在给定位置解析空间描述的指令。为了使更广泛的研究社区能够有效地处理Touchdown任务,我们公开发布了Touchdown所需的29k原始街景照片。我们遵循用于StreetLearn数据发布的过程(Mirowski等人,2019年),以检查个人身份信息和模糊他们必要的。这些数据已添加到StreetLearn数据集,可以通过与StreetLearn之前使用的相同过程获得。我们还提供了两个着陆任务的参考实现:视觉和语言导航(VLN)和空间描述分辨率(SDR)。我们将我们的模型结果与Chen et al.(2019)中给出的结果进行了比较,结果表明,我们添加到StreetLearn中的算法完全支持两种着陆任务,并且可以有效地用于进一步的研究和比较。