arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.19442cs.CV
收藏

UrbanAlign: 域内任务中VLM与人类偏好对齐的后处理语义校准

UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment

Tsinghua University(清华大学);University College London(伦敦大学学院);Hong Kong University of Science and Technology(香港科学与技术大学);Peking University(北京大学);Southwest Jiaotong University(西南交通大学);Zhejiang University(浙江大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Renmin University of China(中国人民大学);Southeast University(东南大学)

Yecheng Zhang, Rong Zhao, Zhizhou Sha, Yong Li, Lei Wang, Ce Hou, Wen Ji, Hao Huang, Yunshan Wan, Jian Yu, Junhao Xia, Yuru Zhang, Chunlei Shi

AI总结:

UrbanAlign通过后处理方法实现VLM与人类偏好的对齐,无需修改模型权重,提升领域任务的准确性和可解释性。

摘要:

视觉语言模型(VLMs)能够对城市场景进行详尽描述,但在特定领域任务如安全评估和审美评价中,始终无法产生可靠的用户偏好标签。常规解决方法,如微调或RLHF,需要大规模标注和模型重训练。我们提出一个不同问题:在不修改任何权重的情况下,能否将冻结的VLM与人类偏好对齐?我们的核心发现是VLMs是强大的概念提取器,但决策校准能力较差。我们提出一个三阶段后处理流程,利用这种不对称性:(i)可解释的评估维度自动从共识示例中挖掘;(ii)Observer-Debater-Judge链从冻结的VLM中提取稳健的概念分数;(iii)在混合流形上进行局部加权岭回归校准这些分数以适应人类评分。在Place Pulse 2.0数据集上应用UrbanAlign框架,框架在六个感知类别上达到72.2%的准确率(kappa=0.45),优于所有基线方法+11.0个百分点,优于零样本VLM+15.5个百分点,具有完全的可解释性和零权重修改。

英文摘要:

Vision-language models (VLMs) can describe urban scenes in rich detail, yet consistently fail to produce reliable human preference labels in domain-specific tasks such as safety assessment and aesthetic evaluation. The standard fix, fine-tuning or RLHF, requires large-scale annotations and model retraining. We ask a different question: can a frozen VLM be aligned with human preferences without modifying any weights? Our key insight is that VLMs are strong concept extractors but poor decision calibrators. We propose a three-stage post-hoc pipeline that exploits this asymmetry: (i) interpretable evaluation dimensions are automatically mined from consensus exemplars; (ii) an Observer-Debater-Judge chain extracts robust concept scores from the frozen VLM; and (iii) locally-weighted ridge regression on a hybrid manifold calibrates these scores to human ratings. Applied as UrbanAlign on Place Pulse 2.0, the framework reaches 72.2% accuracy (kappa=0.45) across six perception categories, outperforming all baselines by +11.0 pp and zero-shot VLM by +15.5 pp, with full interpretability and zero weight modification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑