图像翻译具有广泛的应用,例如风格迁移和模态转换,通常旨在生成兼具高度真实感和保真度的图像。这些问题仍然具有挑战性,特别是在需要保留内容结构时尤为重要。传统的图像级相似性度量作用有限,因为图像的内容结构属于高层信息,并不强烈受制于与原图的像素级保真度。为了填补这一空白,我们引入了SAMScore,这是一种通用的内容结构相似性度量,用于评估图像翻译模型的保真度。SAMScore基于近期高性能的Segment Anything Model (SAM),能够以突出的准确度进行内容相似性比较。我们在19个图像翻译任务上应用了SAMScore,发现它能够在所有任务上优于所有其他竞争性度量。我们设想SAMScore将成为一种有价值的工具,通过允许对新兴和不断演进的翻译模型进行更精确的评估,推动图像翻译这一活跃领域的发展。代码可在https://github.com/Kent0n-Li/SAMScore获取。
英文摘要:
Image translation has wide applications, such as style transfer and modality conversion, usually aiming to generate images having both high degrees of realism and faithfulness. These problems remain difficult, especially when it is important to preserve content structures. Traditional image-level similarity metrics are of limited use, since the content structures of an image are high-level, and not strongly governed by pixel-wise faithfulness to an original image. To fill this gap, we introduce SAMScore, a generic content structural similarity metric for evaluating the faithfulness of image translation models. SAMScore is based on the recent high-performance Segment Anything Model (SAM), which allows content similarity comparisons with standout accuracy. We applied SAMScore on 19 image translation tasks, and found that it is able to outperform all other competitive metrics on all tasks. We envision that SAMScore will prove to be a valuable tool that will help to drive the vibrant field of image translation, by allowing for more precise evaluations of new and evolving translation models. The code is available at https://github.com/Kent0n-Li/SAMScore.
Specular-to-Diffuse Translation for Multi-View Reconstruction
University of Bern(伯尔尼大学);Shenzhen University(深圳大学);Technion - Israel Institute of Technology(以色列理工学院);Tel-Aviv University(特拉维夫大学);University of Maryland(马里兰大学)
Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning
Department of Mechanical and Aerospace Engineering(机械与航空航天工程系);University of Houston(休斯顿大学);School of Engineering(工程学院);Santa Clara University(圣克拉拉大学);School of Electrical and Computer Engineering(电气与计算机工程学院);Cornell University(康奈尔大学);Department of Electrical and Computer Engineering(电气与计算机工程系);Northeastern University(东北大学)
Registration of 3D Point Sets Using Correntropy Similarity Matrix
University of Nevada, Reno(内华达大学雷诺分校);University of Prince Edward Island(爱德华王子岛大学);Japan Advanced Institute of Science and Technology (JAIST)(日本北陆先端科学技术大学院大学(JAIST))