Ancient artifacts are an important medium for cultural preservation and restoration. However, many physical copies of artifacts are either damaged or lost, leaving a blank space in archaeological and historical studies that calls for artifact image generation techniques. Despite the significant advancements in open-domain text-to-image synthesis, existing approaches fail to capture the important domain knowledge presented in the textual description, resulting in errors in recreated images such as incorrect shapes and patterns. In this paper, we propose a novel knowledge-aware artifact image synthesis approach that brings lost historical objects accurately into their visual forms. We use a pretrained diffusion model as backbone and introduce three key techniques to enhance the text-to-image generation framework: 1) we construct prompts with explicit archaeological knowledge elicited from large language models (LLMs); 2) we incorporate additional textual guidance to correlated historical expertise in a contrastive manner; 3) we introduce further visual-semantic constraints on edge and perceptual features that enable our model to learn more intricate visual details of the artifacts. Compared to existing approaches, our proposed model produces higher-quality artifact images that align better with the implicit details and historical knowledge contained within written documents, thus achieving significant improvements across automatic metrics and in human evaluation. Our code and data are available at https://github.com/danielwusg/artifact_diffusion.
AlignGemini: Generalizable AI-Generated Image Detection Through Task-Model Alignment
Tencent Youtu Lab；East China University of Science；Shenzhen University；Hong Kong University of Science；Department of XXX, University of YYY, Location, Country；School of ZZZ, Institute of WWW, Location, Country
Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；INEEJI
Despite significant improvements on the image generation performance of Generative Adversarial Networks (GANs), generations with low visual fidelity still have been observed. As widely used metrics for GANs focus more on the overall performance of the model, evaluation on the quality of individual generations or detection of defective generations is challenging. While recent studies try to detect featuremap units that cause artifacts and evaluate individual samples, these approaches require additional resources such as external networks or a number of training data to approximate the real data…
From Prompting to Composing: A Spatial Canvas Interface for Poster Generation
Fudan University（复旦大学）；Microsoft Research（微软研究院）；The University of Sydney（悉尼大学）；USTC（中国科学技术大学）；University of Waterloo（滑铁卢大学）；The University of Hong Kong（香港大学）
University of Rochester（罗切斯特大学）；University of Maryland, College Park（马里兰大学帕克分校）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）
Current deep neural network based approaches to computed tomography (CT) metal artifact reduction (MAR) are supervised methods that rely on synthesized metal artifacts for training. However, as synthesized data may not accurately simulate the underlying physical mechanisms of CT imaging, the supervised methods often generalize poorly to clinical applications. To address this problem, we propose, to the best of our knowledge, the first unsupervised learning approach to MAR. Specifically, we introduce a novel artifact disentanglement network that disentangles the metal artifacts from CT images i…
CoEvoT: Co-Evolving Chain-of-Thought Prompting for Graph-LLM Reasoning
Sun Yat-Sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；National University of Singapore（新加坡国立大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Posts and Telecommunieations（北京邮电大学）；Singapore Management University（新加坡管理大学）
Harbin Institute of Technology（哈尔滨工业大学）；University of Burgundy Franche-Comté（勃艮第-弗朗什-孔泰大学）；University of the Basque Country（巴斯克大学）；Anyvision；Peng Cheng Lab（鹏城实验室）
Despite recent advances in deep learning-based face frontalization methods, photo-realistic and illumination preserving frontal face synthesis is still challenging due to large pose and illumination discrepancy during training. We propose a novel Flow-based Feature Warping Model (FFWM) which can learn to synthesize photo-realistic and illumination preserving frontal images with illumination inconsistent supervision. Specifically, an Illumination Preserving Module (IPM) is proposed to learn illumination preserving image synthesis from illumination inconsistent image pairs. IPM includes two path…
ACD: Direct Conditional Control for Video Diffusion Models via Attention Supervision
Sun Yat-sen University（中山大学）；Yonsei University（延世大学）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）；X-Era AI Lab（X-Era人工智能实验室）
LEAVS: An LLM-based Labeler for Abdominal CT Supervision
National Institutes of Health Clinical Center（美国国家卫生研究院临床中心）；Icahn School of Medicine at Mount Sinai（伊坎医学院）；Walter Reed National Military Medical Center（沃尔特·里德国家军事医疗中心）
RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing
Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））；The Chinese University of Hong Kong (CUHK)（香港中文大学）；Sun Yat-sen University（中山大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）
Our work explores temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationships in the generated data are much less explored. Natural temporal changes are crucial for sequential generation tasks, e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as $L^2$ over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail…
Surrogate Supervision for Robust and Generalizable Deformable Image Registration
Department of Electrical and Computer Engineering, Vanderbilt University（维斯尼尔大学电气与计算机工程系）；Department of Radiology and Radiological Science, Johns Hopkins Medical School（约翰霍普金斯医学学校放射学与放射科学系）；Image Analysis and Communications Laboratory in the Department of Electrical and Computer Engineering, Johns Hopkins University（约翰霍普金斯大学电气与计算机工程系图像分析与通信实验室）；Center for Cognitive Medicine, Department of Psychiatry and Behavioral Science, Vanderbilt University Medical Center（维斯尼尔大学医学中心认知医学中心）；University of Pittsburgh, School of Medicine（匹兹堡大学医学院）
Recent progress has shown that large-scale pre-training using contrastive image-text pairs can be a promising alternative for high-quality visual representation learning from natural language supervision. Benefiting from a broader source of supervision, this new paradigm exhibits impressive transferability to downstream classification tasks and datasets. However, the problem of transferring the knowledge learned from image-text pairs to more complex dense prediction tasks has barely been visited. In this work, we present a new framework for dense prediction by implicitly and explicitly leverag…
Renmin University of China（中国人民大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
The key challenge of image manipulation detection is how to learn generalizable features that are sensitive to manipulations in novel data, whilst specific to prevent false alarms on authentic images. Current research emphasizes the sensitivity, with the specificity overlooked. In this paper we address both aspects by multi-view feature learning and multi-scale supervision. By exploiting noise distribution and boundary artifact surrounding tampered regions, the former aims to learn semantic-agnostic and thus more generalizable features. The latter allows us to learn from authentic images which…
Learning 3D Medical Image Models From Brain Functional Connectivity Network Supervision For Mental Disorder Diagnosis
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）；Chinese Academy of Sciences（中国科学院）
Photoacoustic (PA) computed tomography (PACT) reconstructs the initial pressure distribution from raw PA signals. The standard reconstruction of medical image could cause the artifacts due to interferences or ill-posed setup. Recently, deep learning has been used to reconstruct the PA image with ill-posed conditions. Most works remove the artifacts from image domain, and compensate the limited-view from dataset. In this paper, we propose a jointed feature fusion framework (JEFF-Net) based on deep learning to reconstruct the PA image using limited-view data. The cross-domain features from limit…
Few-shot action recognition in videos is challenging for its lack of supervision and difficulty in generalizing to unseen actions. To address this task, we propose a simple yet effective method, called knowledge prompting, which leverages commonsense knowledge of actions from external resources to prompt a powerful pre-trained vision-language model for few-shot classification. We first collect large-scale language descriptions of actions, defined as text proposals, to build an action knowledge base. The collection of text proposals is done by filling in handcraft sentence templates with extern…
SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；University of Oxford（牛津大学）；YouTu Lab, Tencent（腾讯优图实验室）
Advancing Weakly-Supervised Change Detection in Satellite Images via Adversarial Class Prompting
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（信息工程测绘遥感国家重点实验室，武汉大学）；School of Computer Science and Technology, Anhui University（计算机科学与技术学院，安徽大学）；Graduate School of Frontier Sciences, University of Tokyo（前沿科学研究院，东京大学）；Institute of Geodesy and Photogrammetry, ETH Zürich（测绘学研究院，苏黎世联邦理工学院）；Department of Computer Science, Stanford University（计算机科学系，斯坦福大学）
Johns Hopkins University（约翰斯·霍普金斯大学）；National Institute on Aging, National Institutes of Health（美国国立卫生研究院国家衰老研究所）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）
Image quality control (IQC) can be used in automated magnetic resonance (MR) image analysis to exclude erroneous results caused by poorly acquired or artifact-laden images. Existing IQC methods for MR imaging generally require human effort to craft meaningful features or label large datasets for supervised training. The involvement of human labor can be burdensome and biased, as labeling MR images based on their quality is a subjective task. In this paper, we propose an automatic IQC method that evaluates the extent of artifacts in MR images without supervision. In particular, we design an art…
Deep neural network based methods have achieved promising results for CT metal artifact reduction (MAR), most of which use many synthesized paired images for training. As synthesized metal artifacts in CT images may not accurately reflect the clinical counterparts, an artifact disentanglement network (ADN) was proposed with unpaired clinical images directly, producing promising results on clinical datasets. However, without sufficient supervision, it is difficult for ADN to recover structural details of artifact-affected CT images based on adversarial losses only. To overcome these problems, h…
Foundation Model-guided Iteratively Prompting and Pseudo-Labeling for Partially Labeled Medical Image Segmentation
Department of Biomedical Engineering, Columbia University（哥伦比亚大学生物医学工程系）；Department of Radiation Oncology, Columbia University Irving Medical Center（哥伦比亚大学欧文医学中心放射肿瘤科）；Data Science Institute, Columbia University（哥伦比亚大学数据科学研究所）；Herbert Irving Comprehensive Cancer Center, Columbia University（哥伦比亚大学赫伯特·欧文综合癌症中心）；Department of Radiation Oncology, Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院放射肿瘤科）
Generalizable Deepfake Detection Based on Forgery-aware Layer Masking and Multi-artifact Subspace Decomposition
Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology（南京信息工程大学数字取证教育部工程研究中心）
