MaterialPicker: Multi-Modal DiT-Based Material Generation
State Key Lab of CAD\&CG, Zhejiang University（浙江大学CAD与CG国家重点实验室）；Adobe Research（Adobe研究）；State Key Lab of CAD\&CG, Zhejiang University and ZJU-FaceUnity Joint Lab of Intelligent Graphics（浙江大学CAD与CG国家重点实验室）；ZJU-FaceUnity Joint Lab of Intelligent Graphics（浙大-面 unity 智能图形联合实验室）
Xiaohe Ma, Valentin Deschaintre, Miloš Hašan, Fujun Luan, Kun Zhou, Hongzhi Wu, Yiwei Hu
本文提出 MaterialPicker，通过微调基于 DiT 的视频生成器，将材质贴图视为视频帧，支持文本与照片多模态输入，可从变形、倾斜或遮挡的材质照片中生成更多样、校正更好的高质量材质。
摘要：
高质量材质生成是虚拟环境创作与逆向渲染的关键。我们提出 MaterialPicker，这是一种利用 Diffusion Transformer（DiT，扩散 Transformer）架构的多模态材质生成器，改进并简化了根据文本提示和/或照片创建高质量材质的流程。我们的方法能够基于材质样本的图像裁剪生成材质，即使被拍摄表面发生变形、以倾斜角度呈现或被部分遮挡也不受影响，而这些情况在自然场景照片中十分常见。我们还允许用户指定文本提示，为生成提供额外引导。我们将预训练的基于 DiT 的视频生成器微调为材质生成器，其中每张材质贴图都被视为视频序列中的一帧。我们对该方法进行了定量与定性评估，结果表明，与以往工作相比，它能够实现更加多样化的材质生成以及更好的变形校正。
英文摘要：
High-quality material generation is key for virtual environment authoring and inverse rendering. We propose MaterialPicker, a multi-modal material generator leveraging a Diffusion Transformer (DiT) architecture, improving and simplifying the creation of high-quality materials from text prompts and/or photographs. Our method can generate a material based on an image crop of a material sample, even if the captured surface is distorted, viewed at an angle or partially occluded, as is often the case in photographs of natural scenes. We further allow the user to specify a text prompt to provide additional guidance for the generation. We finetune a pre-trained DiT-based video generator into a material generator, where each material map is treated as a frame in a video sequence. We evaluate our approach both quantitatively and qualitatively and show that it enables more diverse material generation and better distortion correction than previous work.
DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation
MMLab, CUHK（香港中文大学多媒体实验室）；GVC Lab, Great Bay University（大湾区大学GVC实验室）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；Tencent AI Lab（腾讯AI实验室）；SHIAE, CUHK（香港中文大学SHIAE实验室）
Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers
The University of Hong Kong（香港大学）；Nanjing University（南京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Nanyang Technological University（南洋理工大学）；Harbin Institute of Technology（哈尔滨工业大学）
CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation
Institute of Trustworthy Embodied AI（可信具身人工智能研究院）；Fudan University（复旦大学）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Bytedance Intelligent Creation（字节跳动智能创作）
GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers
Codec Avatars Lab, Meta（Meta编码器动画实验室）；University of Tübingen（图宾根大学）；Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息学院，萨尔兰信息校园）
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；Tencent Hunyuan（腾讯文英）；Shanghai Jiao Tong University（上海交通大学）；Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Shanghai AI Lab（上海人工智能实验室）
BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration
University of Science and Technology of China（中国科学技术大学）；ByteDance（字节跳动）；National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory（国家深空探测重点实验室，深空探测实验室）
Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beihang University（北京航空航天大学）；Nanyang Technological University（南洋理工大学）
ORCA: Hunting Compositional Failures in Text-to-Image Diffusion
Wellcome Sanger Institute（威康桑格研究所）；Cambridge Stem Cell Institute, University of Cambridge（剑桥大学剑桥干细胞研究所）；Cambridge Centre for AI in Medicine, University of Cambridge（剑桥大学剑桥人工智能医学中心）；University of Oxford（牛津大学）；Apple（苹果公司）；University of Helsinki（赫尔辛基大学）；Institute for Molecular Medicine Finland (FIMM), University of Helsinki（赫尔辛基大学芬兰分子医学研究所）；School of Computing and Information Systems, University of Melbourne（墨尔本大学计算与信息系统学院）；Department of Medicine, University of Cambridge（剑桥大学医学系）
Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement
Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程系，中国沈阳）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University, Shenyang, China（教育部医学图像智能计算重点实验室，东北大学，中国沈阳）；National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, China（工业智能与系统优化国家级前沿科学中心，中国沈阳）；AiShiWeiLai AI Research, China（艾世维来人工智能研究，中国）；Amii, University of Alberta, Edmonton, Alberta, Canada（阿尔伯塔大学艾米人工智能研究所，加拿大埃德蒙顿，阿尔伯塔）
Training-Free Text-Guided Color Editing with Multi-Modal Diffusion Transformer
The Hong Kong University of Science and Technology（香港科学与技术大学）；International Digital Economy Academy（国际数字经济学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Tsinghua University（清华大学）；Astribot；StepFun
Diff-Feat: Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification
School of Statistics, Renmin University of China（中国人民大学统计学院）；Center for Applied Statistics and School of Statistics, Renmin University of China（中国人民大学应用统计中心和统计学院）
UTDesign: A Unified Framework for Stylized Text Editing and Generation in Graphic Design Images
Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Lumina-Image 2.0: A Unified and Efficient Image Generative Framework
Shanghai AI Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Jiao Tong University（上海交通大学）；Krea AI（Krea AI公司）
UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors
Brown University（布朗大学）；Massachusetts Institute of Technology（麻省理工学院）；Meta Reality Lab（Meta现实实验室）；Max-Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；University of Hong Kong（香港大学）
