OrthoGeoLoRA:用于Web上结构化社会科学概念检索的几何参数高效微调
OrthoGeoLoRA: Geometric Parameter-Efficient Fine-Tuning for Structured Social Science Concept Retrieval on theWeb
AI总结:
OrthoGeoLoRA通过几何约束改进低秩适应,实现更高效的参数微调,提升社会科学概念检索性能。
摘要:
大型语言模型和文本编码器越来越多地用于社会科学中的网络信息系统,包括数字图书馆、数据目录和研究人员、政策制定者和社会公众使用的搜索界面。完全微调通常计算上和能源上都很昂贵,这可能对Web4Good生态系统中的小型机构和非营利组织具有限制性。参数高效微调(PEFT),尤其是低秩适应(LoRA),通过仅更新少量参数来降低这种成本。我们证明了标准LoRA更新ΔW=BA^T存在几何缺点:规范自由度、尺度模糊性和倾向于秩崩溃。我们引入了OrthoGeoLoRA,通过约束低秩因子为正交(Stiefel流形)来强制采用SVD-like形式ΔW=BΣA^T。几何再参数化实现了这一约束,同时与Adam等标准优化器和现有微调流程保持兼容。我们还提出了一个基准,用于在欧洲语言社会科学主题词表(ELSST)上进行层次概念检索,该词表广泛用于组织数字存储库中的社会科学资源。使用多语言句子编码器的实验表明,OrthoGeoLoRA在相同低秩预算下优于标准LoRA和几种强大的PEFT变体,在排名指标上表现更优,为资源受限环境下的基础模型适应提供了一条更计算和参数高效的道路。
英文摘要:
Large language models and text encoders increasingly power web-based information systems in the social sciences, including digital libraries, data catalogues, and search interfaces used by researchers, policymakers, and civil society. Full fine-tuning is often computationally and energy intensive, which can be prohibitive for smaller institutions and non-profit organizations in the Web4Good ecosystem. Parameter-Efficient Fine-Tuning (PEFT), especially Low-Rank Adaptation (LoRA), reduces this cost by updating only a small number of parameters. We show that the standard LoRA update $ΔW = BA^\top$ has geometric drawbacks: gauge freedom, scale ambiguity, and a tendency toward rank collapse. We introduce OrthoGeoLoRA, which enforces an SVD-like form $ΔW = BΣA^\top$ by constraining the low-rank factors to be orthogonal (Stiefel manifold). A geometric reparameterization implements this constraint while remaining compatible with standard optimizers such as Adam and existing fine-tuning pipelines. We also propose a benchmark for hierarchical concept retrieval over the European Language Social Science Thesaurus (ELSST), widely used to organize social science resources in digital repositories. Experiments with a multilingual sentence encoder show that OrthoGeoLoRA outperforms standard LoRA and several strong PEFT variants on ranking metrics under the same low-rank budget, offering a more compute- and parameter-efficient path to adapt foundation models in resource-constrained settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。