We systematically investigate the parameter-efficient fine-tuning design space under practical data and compute constraints, and propose D2-LoRA. D2-LoRA achieves 76.4 percent average accuracy across eight question answering and reading comprehension benchmarks using only 5k training samples per task and two epochs, while preserving algebraic mergeability at inference with near-exact numerical equivalence. The method combines signed low-rank residual updates with additive and subtractive components, together with a train-time column-wise projection that keeps each column close to its original norm. After training, the adapter is merged into a single weight matrix, adding zero inference latency. Compared with LoRA, D2-LoRA improves average accuracy by 2.2 percentage points; at matched parameter counts (LoRA rank 2r versus D2-LoRA rank r), the improvement is 1.6 points, indicating gains from architectural design rather than increased parameterization. Compared with DoRA, it matches or exceeds performance on most tasks. Beyond QA and reading comprehension, D2-LoRA improves generative tasks (plus 1.2 ROUGE-L and plus 1.1 percent win rate) and shows 36 percent lower training volatility. The merge preserves numerical fidelity (mean gap about 0.03 percentage points) and recovers about 1.91x evaluation throughput. Training overhead is 19 percent, comparable to DoRA, and decreases with longer input sequences. We provide a geometric analysis explaining how the projection stabilizes training, together with ablation studies isolating the contribution of each design component.
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（图像处理与模式识别研究所，自动化与智能感知学院，上海交通大学）；MoE Key Laboratory of System Control and Information Processing (Shanghai)（系统控制与信息处理MOE重点实验室（上海））
DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Science, VNU-HCM（胡志明市国家大学科学大学）；Vietnam National University, Ho Chi Minh City（胡志明市越南国家大学）；Trivita AI
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence（北京交通大数据挖掘与具身智能重点实验室）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）；State Key Laboratory of Advanced Rail Autonomous Operation（先进轨道交通自主运行国家重点实验室）
