基于分解式评估的人类偏好对齐概念定制基准
Human Preference-Aligned Concept Customization Benchmark via Decomposed Evaluation
AI总结：
针对现有概念定制评估指标与人类偏好不对齐的问题，提出分解式GPT评分方法，同时发布涵盖单/多概念任务的CC-AlignBench基准，该方法与人类偏好相关性更高，为概念定制评估建立了新标准。
摘要：
概念定制的评估颇具挑战性，因为它需要全面评估生成结果对生成提示词和概念图像的保真度。此外，多概念评估比单概念评估难度大得多，因为它不仅需要对每个单独概念进行细致评估，还要考量概念间的相互作用。虽然人类能够直观地评估生成的图像，但现有指标给出的评估往往要么过于局限，要么过于泛化，导致与人类偏好不对齐。为解决这一问题，我们提出了分解式GPT评分（Decomposed GPT Score，D-GPTScore），这是一种新型的人类对齐评估方法，它将评估标准分解为更细的维度，并利用多模态大语言模型（Multimodal Large Language Model，MLLM）开展分维度评估。此外，我们发布了人类偏好对齐概念定制基准（Human Preference-Aligned Concept Customization Benchmark，CC-AlignBench），这一基准数据集同时包含单概念和多概念任务，可支持从单个动作到多人交互的宽难度范围的分阶段评估。我们的方法在该基准上显著优于现有方法，与人类偏好的相关性更高。这项工作为概念定制评估建立了新标准，也点明了未来研究的关键挑战。该基准及相关材料可在https://github.com/ReinaIshikawa/D-GPTScore获取。
英文摘要：
Evaluating concept customization is challenging, as it requires a comprehensive assessment of fidelity to generative prompts and concept images. Moreover, evaluating multiple concepts is considerably more difficult than evaluating a single concept, as it demands detailed assessment not only for each individual concept but also for the interactions among concepts. While humans can intuitively assess generated images, existing metrics often provide either overly narrow or overly generalized evaluations, resulting in misalignment with human preference. To address this, we propose Decomposed GPT Score (D-GPTScore), a novel human-aligned evaluation method that decomposes evaluation criteria into finer aspects and incorporates aspect-wise assessments using Multimodal Large Language Model (MLLM). Additionally, we release Human Preference-Aligned Concept Customization Benchmark (CC-AlignBench), a benchmark dataset containing both single- and multi-concept tasks, enabling stage-wise evaluation across a wide difficulty range -- from individual actions to multi-person interactions. Our method significantly outperforms existing approaches on this benchmark, exhibiting higher correlation with human preferences. This work establishes a new standard for evaluating concept customization and highlights key challenges for future research. The benchmark and associated materials are available at https://github.com/ReinaIshikawa/D-GPTScore.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
