千言万语或一幅图像：研究多模态大语言模型中人物角色模态的影响
A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMs
AI总结：
本研究探讨多模态LLM中人物角色模态（文本、图像等）对体现效果的影响，构建40个角色的平行数据集和60问评估框架，发现文本更显语言习惯，图像易被忽略，为改进提供方向。
摘要：
大型语言模型（LLMs）最近在体现多样化人物角色方面展现出显著进步，增强了其作为对话代理和虚拟助手的有效性。因此，LLMs在处理和整合多模态信息方面取得了重大进展。然而，尽管人类人物角色可以通过文本和图像两种方式表达，但人物角色的模态对LLM体现程度的影响在很大程度上仍未得到探索。在本文中，我们研究了不同模态如何影响多模态LLMs中人物角色的表现力。为此，我们创建了一个新颖的模态平行数据集，包含40个多样化的人物角色，涵盖年龄、性别、职业和地点等差异。该数据集包含四种模态来等价地表示一个人物角色：仅图像、仅文本、图像与小文本的组合，以及排版图像（其中文本以视觉风格化方式呈现以传达人物角色相关属性）。然后，我们构建了一个系统评估框架，包含60个问题和相应指标，以评估LLMs在属性和场景中体现每个人物角色的程度。对5个多模态LLMs的综合实验表明，由详细文本表示的人物角色展现出更多的语言习惯，而排版图像通常显示出与人物角色更高的一致性。我们的结果揭示，LLMs常常忽略通过图像传达的人物角色特定细节，突出了潜在局限性，并为未来研究弥合这一差距铺平了道路。我们在https://github.com/claws-lab/persona-modality 发布了数据和代码。
英文摘要：
Large language models (LLMs) have recently demonstrated remarkable advancements in embodying diverse personas, enhancing their effectiveness as conversational agents and virtual assistants. Consequently, LLMs have made significant strides in processing and integrating multimodal information. However, even though human personas can be expressed in both text and image, the extent to which the modality of a persona impacts the embodiment by the LLM remains largely unexplored. In this paper, we investigate how do different modalities influence the expressiveness of personas in multimodal LLMs. To this end, we create a novel modality-parallel dataset of 40 diverse personas varying in age, gender, occupation, and location. This consists of four modalities to equivalently represent a persona: image-only, text-only, a combination of image and small text, and typographical images, where text is visually stylized to convey persona-related attributes. We then create a systematic evaluation framework with 60 questions and corresponding metrics to assess how well LLMs embody each persona across its attributes and scenarios. Comprehensive experiments on $5$ multimodal LLMs show that personas represented by detailed text show more linguistic habits, while typographical images often show more consistency with the persona. Our results reveal that LLMs often overlook persona-specific details conveyed through images, highlighting underlying limitations and paving the way for future research to bridge this gap. We release the data and code at https://github.com/claws-lab/persona-modality .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
