HINT：面向高效零样本与少样本泛化的超网络指令微调
HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation
AI总结：
HINT提出用超网络将任务指令和示例转换为参数高效模块，在解码时拼接编码指令，从而以极低额外计算成本实现高效零样本与少样本泛化，性能显著超越现有基线。
摘要：
近期自然语言处理模型展现出仅以自然语言指令为引导即可有效泛化到新任务的“零样本”能力。然而，许多此类方法因依赖将冗长指令与每个输入样本拼接而导致高昂的计算成本，造成指令被重复处理。为避免这一问题，我们提出超网络指令微调（HINT），该方法利用预训练文本编码器将任务指令和示例转换为参数高效模块，并插入到底层模型中，从而无需在模型输入中包含指令。HINT中的超网络还会生成一个编码后的指令，在解码阶段将其与编码后的输入拼接，以进一步提升性能。在控制计算量（以FLOPs衡量）的条件下，HINT模型比强基线的最先进方法性能高出10%以上。通过将指令转换为模块，HINT模型在计算使用上可以有效忽略指令长度和少样本示例输入的长度。因此，HINT可以通过引入额外的少样本数据将性能提升最多25%，而计算量仅增加最多5%。这一方法结合了参数高效微调与上下文学习的优势。
英文摘要：
Recent NLP models have shown the remarkable ability to effectively generalise `zero-shot' to new tasks using only natural language instructions as guidance. However, many of these approaches suffer from high computational costs due to their reliance on concatenating lengthy instructions with every input example, resulting in costly reprocessing of the instruction. To avoid this, we introduce Hypernetworks for INstruction Tuning (HINT), which convert task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder, eliminating the need to include instructions in the model input. The hypernetwork in HINT also produces an encoded instruction, which we concatenate with encoded inputs during decoding to further improve performance. HINT models outperform strong state-of-the-art baselines by over 10% when controlling for compute (measured in FLOPs). By converting instructions into modules, HINT models can effectively disregard the length of instructions and few-shot example inputs in terms of compute usage. As a result, HINT can enhance its performance by up to 25% by incorporating additional few-shot data, while utilizing only up to 5% more compute. This combines the strengths of parameter-efficient fine-tuning and in-context learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
