AI 中文总结
研究针对LoRA方法中秩分配问题，提出IFCLoRA拓扑感知秩分配方法，利用小校准集和冻结模型构建交互图，结合全局拓扑先验与局部梯度敏感性计算得分来分配秩，在多场景下优于其他方法，还揭示了任务依赖的秩分布特点。
AI 中文摘要
低秩适应（LoRA）是一种广泛用于大语言模型的参数高效微调方法，但其性能很大程度上取决于固定秩预算如何在Transformer模块间分配。现有自适应秩方法通常依赖训练时收集的局部梯度统计，这会引入额外内存和计算并忽略任务条件全局信息流。我们提出IFCLoRA，一种在微调前应用的拓扑感知秩分配方法。利用小校准集和冻结预训练模型，构建稀疏任务条件交互图，其节点代表LoRA兼容模块。结合全局信息流拓扑先验和局部梯度敏感性计算信息流中心性得分，估计各模块在多跳传播下的适应重要性，然后在全局预算下一次性分配秩。在多个模型、任务和低秩设置中，IFCLoRA在匹配训练配置和总秩预算下始终优于LoRA、AdaLoRA和EVA，同时保持与标准LoRA相当的训练成本。在使用LLaMA 3 8B进行数学推理时，IFCLoRA在秩4时比LoRA提高1.36%，秩8时提高1.82%。进一步分析表明存在任务依赖的非均匀秩分布，这表明全局信息流结构为低预算参数高效微调提供了信息丰富且可解释的先验。
英文摘要
Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT) of LLMs whose effectiveness depends on rank allocation. Existing adaptive LoRA methods derive ranks from local gradient, activation, or matrix statistics collected before or during fine-tuning; training-time variants add overhead, and local signals reveal little about each module's structural role in information propagation, giving weak global grounding for scarce-capacity allocation. We propose IFCLoRA, a topology-aware method for pre-fine-tuning rank allocation and adapter initialization. Using a small calibration set, IFCLoRA performs intervention tracing on the frozen model and constructs a sparse task-conditioned interaction graph over LoRA target modules. From this graph it extracts a global information-flow topology prior and fuses it with each node's local gradient sensitivity to form a topology-dominant Information-Flow Centrality (IFC) score, measuring participation in task-conditioned multi-hop propagation. The IFC scores then serve as module-level routing signals for one-shot discrete rank allocation under a rank-budget constraint. Reusing response vectors from tracing, IFCLoRA constructs a function-preserving flow-response subspace initialization, giving adapters task-relevant output subspaces. Across all settings, IFCLoRA achieves higher mean scores than standard LoRA with comparable fine-tuning time and peak memory; it requires a one-time offline calibration stage. On GSM8K, IFCLoRA attains the highest mean accuracy among compared PEFT methods on both base models, exceeding standard LoRA by 4.75 percentage points on LLaMA-3.1-8B. Resulting rank allocations are non-uniform and vary across tasks and base models, suggesting that task-conditioned global information-flow topology can serve as a useful structural prior for rank allocation in low-budget PEFT.
Comments9 pages, 5 figures
