微调因果大语言模型用于文本分类：基于嵌入与基于指令的方法
Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches
AI总结：
本文探索在资源受限下微调解码器-only大语言模型用于文本分类，比较了基于嵌入的分类头方法和基于指令的微调方法，并采用4位量化与LoRA实现高效训练，实验表明嵌入头方法在单标签分类中匹配或超越微调BERT基线，而指令微调仅在多标签且大参数量时有效。
摘要：
我们探索在资源受限下高效微调解码器-only大语言模型（LLMs）用于下游文本分类的策略。研究了两种方法：(1) 将分类头附加到预训练的因果LLM上，并在任务上微调，使用LLM的最终token嵌入作为序列表示；(2) 以提示-响应的格式对LLM进行指令微调以进行分类。为了在单GPU上微调高达8B参数的模型，我们将4位模型量化与低秩适配（LoRA）结合，实现参数高效训练。在两个专利基准测试（一个5类单标签内部语料库和具有14个类别的公共WIPO-Alpha多标签数据集）上的实验表明，嵌入头方法在单标签分类中匹配或超过微调BERT基线，同时训练参数少10-30倍。指令微调仅在多标签场景下具有竞争力，且需要至少1亿参数的大幅可训练预算。这些结果表明，直接利用因果LLM的内部表示，结合高效微调技术，在有限计算资源下能产生强大的分类性能。我们讨论了每种方法的优势，并概述了在分类场景中优化LLM微调的实用指南和未来方向。
英文摘要：
We explore efficient strategies to fine-tune decoder-only Large Language Models (LLMs) for downstream text classification under resource constraints. Two approaches are investigated: (1) attaching a classification head to a pretrained causal LLM and fine-tuning it on the task, using the LLM's final-token embedding as a sequence representation, and (2) instruction-tuning the LLM in a prompt-to-response format for classification. To enable single-GPU fine-tuning of models up to 8B parameters, we combine 4-bit model quantization with Low-Rank Adaptation (LoRA) for parameter-efficient training. Experiments on two patent benchmarks, a 5-class single-label internal corpus and the public WIPO-Alpha multi-label dataset with 14 categories, show that the embedding-head approach matches or exceeds fine-tuned BERT baselines on single-label classification while training 10-30x fewer parameters. Instruction-tuning is competitive only in the multi-label regime, and only with substantially larger trainable budgets of at least 100M parameters. These results demonstrate that directly leveraging the internal representations of causal LLMs, together with efficient fine-tuning techniques, yields strong classification performance under limited computational resources. We discuss the advantages of each approach and outline practical guidelines and future directions for optimizing LLM fine-tuning in classification scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
