arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.15539cs.CVcs.AIcs.SC
收藏

动态免训练 主体与风格LoRA融合

Dynamic Training-Free Fusion of Subject and Style LoRAs

Shanghai Jiao Tong University(上海交通大学);Eastern Institute of Technology(技术东院)

Qinglong Cao, Yuntian Chen, Chao Ma, Xiaokang Yang

AI总结:

本文提出一种动态免训练的LoRA融合框架,通过特征级选择和度量引导的潜在调整实现主体与风格的动态合成,无需重新训练即可提升生成效果。

摘要:

近期研究探讨了结合多个LoRA以同时生成用户指定的主题和风格。然而,大多数现有方法通过静态统计启发式融合LoRA权重,偏离了LoRA最初学习适应性特征调整的目的,并忽略了采样输入的随机性。为此,我们提出了一种动态免训练融合框架,该框架在整个生成过程中运行。在正向传递过程中,每次应用LoRA的层中,我们动态计算基础模型原始特征与主体和风格LoRA生成特征之间的KL散度,并自适应选择最合适的权重进行融合。在反向去噪阶段,我们进一步通过动态应用基于目标度量(如CLIP和DINO分数)的梯度修正来细化生成轨迹,提供连续的语义和风格指导。通过在整个扩散时间线上整合这两种互补机制——特征级选择和度量引导的潜在调整,我们的方法在不进行重新训练的情况下实现了动态的主体-风格合成。在多样化的主体-风格组合上进行的广泛实验表明,我们的方法在定性和定量上均优于最先进的LoRA融合方法。

英文摘要:

Recent studies have explored the combination of multiple LoRAs to simultaneously generate user-specified subjects and styles. However, most existing approaches fuse LoRA weights using static statistical heuristics that deviate from LoRA's original purpose of learning adaptive feature adjustments and ignore the randomness of sampled inputs. To address this, we propose a dynamic training-free fusion framework that operates throughout the generation process. During the forward pass, at each LoRA-applied layer, we dynamically compute the KL divergence between the base model's original features and those produced by subject and style LoRAs, respectively, and adaptively select the most appropriate weights for fusion. In the reverse denoising stage, we further refine the generation trajectory by dynamically applying gradient-based corrections derived from objective metrics such as CLIP and DINO scores, providing continuous semantic and stylistic guidance. By integrating these two complementary mechanisms-feature-level selection and metric-guided latent adjustment-across the entire diffusion timeline, our method dynamically achieves coherent subject-style synthesis without any retraining. Extensive experiments across diverse subject-style combinations demonstrate that our approach consistently outperforms state-of-the-art LoRA fusion methods both qualitatively and quantitatively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑