夏洛克·福尔摩斯之眼：通过视觉语言模型智能体框架揭示用户私有属性画像
The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework
AI总结：
本研究提出HolmesEye混合智能体框架，结合VLM与LLM从个人图像推断私有属性，构建PAPI数据集，准确率较基线提升10.8%，揭示图像画像隐私风险。
摘要：
我们的研究揭示了一种与视觉语言模型（VLM）智能体框架相关的新型隐私风险：从一组个人图像中推断敏感属性（如年龄和健康信息）甚至抽象属性（如性格和社交特质）的能力，我们将其称为“图像私有属性画像”。鉴于现代应用程序可以轻松访问用户的相册，且从图像集合中进行推断使模型能够利用图像间关系进行更复杂的画像，这种威胁尤为严重。然而，两个主要挑战阻碍了我们对VLM从少量个人照片中对个体进行画像的能力的理解：（1）缺乏具有多图像私有属性标注的基准数据集，（2）当前多模态大语言模型（MLLM）从大型图像集合中推断抽象属性的能力有限。在这项工作中，我们构建了PAPI，这是用于研究个人图像中私有属性画像的最大数据集，包含来自251个个体的2,510张图像，并带有3,012个标注的隐私属性。我们还提出了HolmesEye，一种结合VLM和LLM的混合智能体框架，以增强隐私推断。HolmesEye利用VLM提取图像内和图像间信息，并利用LLM指导推断过程以及通过取证分析整合结果，克服了现有长上下文视觉推理的局限性。实验表明，HolmesEye在平均准确率上比最先进的基线提高了10.8%，在预测抽象属性方面超越了人类水平15.0%。这项工作强调了解决基于图像的画像中隐私风险的紧迫性，并提供了新的数据集和先进框架，以指导该领域的未来研究。
英文摘要：
Our research reveals a new privacy risk associated with the vision-language model (VLM) agentic framework: the ability to infer sensitive attributes (e.g., age and health information) and even abstract ones (e.g., personality and social traits) from a set of personal images, which we term "image private attribute profiling." This threat is particularly severe given that modern apps can easily access users' photo albums, and inference from image sets enables models to exploit inter-image relations for more sophisticated profiling. However, two main challenges hinder our understanding of how well VLMs can profile an individual from a few personal photos: (1) the lack of benchmark datasets with multi-image annotations for private attributes, and (2) the limited ability of current multimodal large language models (MLLMs) to infer abstract attributes from large image collections. In this work, we construct PAPI, the largest dataset for studying private attribute profiling in personal images, comprising 2,510 images from 251 individuals with 3,012 annotated privacy attributes. We also propose HolmesEye, a hybrid agentic framework that combines VLMs and LLMs to enhance privacy inference. HolmesEye uses VLMs to extract both intra-image and inter-image information and LLMs to guide the inference process as well as consolidate the results through forensic analysis, overcoming existing limitations in long-context visual reasoning. Experiments reveal that HolmesEye achieves a 10.8% improvement in average accuracy over state-of-the-art baselines and surpasses human-level performance by 15.0% in predicting abstract attributes. This work highlights the urgency of addressing privacy risks in image-based profiling and offers both a new dataset and an advanced framework to guide future research in this area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
