批量校准：重新思考上下文学习与提示工程的校准
Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering
AI总结：
本文针对提示与 ICL 中的上下文偏差问题，提出零样本、仅推理且低成本的 Batch Calibration，并扩展至多任务少样本学习，在 PaLM 2 与 CLIP 上取得最优校准性能。
摘要：
提示和上下文学习（ICL）已成为大型语言模型（LLM）的高效学习范式。然而，LLM 受到提示脆弱性以及提示中各种偏差因素的影响，这些因素包括但不限于格式、言语化器选择以及 ICL 示例。为了解决这一导致意外性能下降的问题，研究者开发了校准方法，以减轻这些偏差的影响并恢复 LLM 的性能。在本研究中，我们首先对现有校准方法进行系统分析，既提供统一视角，也揭示其失败情形。受这些分析启发，我们提出批量校准（Batch Calibration，BC），这是一种简单而直观的方法，通过控制来自批量输入的上下文偏差，统一了多种先前方法，并有效解决上述问题。BC 是零样本、仅推理的方法，并且只会产生可忽略的额外成本。在少样本设置中，我们进一步扩展 BC，使其能够从有标注数据中学习上下文偏差。我们使用 PaLM 2-(S, M, L) 和 CLIP 模型验证了 BC 的有效性，并在超过 10 项自然语言理解和图像分类任务上证明，其相较先前校准基线取得了最先进性能。
英文摘要：
Prompting and in-context learning (ICL) have become efficient learning paradigms for large language models (LLMs). However, LLMs suffer from prompt brittleness and various bias factors in the prompt, including but not limited to the formatting, the choice verbalizers, and the ICL examples. To address this problem that results in unexpected performance degradation, calibration methods have been developed to mitigate the effects of these biases while recovering LLM performance. In this work, we first conduct a systematic analysis of the existing calibration methods, where we both provide a unified view and reveal the failure cases. Inspired by these analyses, we propose Batch Calibration (BC), a simple yet intuitive method that controls the contextual bias from the batched input, unifies various prior approaches, and effectively addresses the aforementioned issues. BC is zero-shot, inference-only, and incurs negligible additional costs. In the few-shot setup, we further extend BC to allow it to learn the contextual bias from labeled data. We validate the effectiveness of BC with PaLM 2-(S, M, L) and CLIP models and demonstrate state-of-the-art performance over previous calibration baselines across more than 10 natural language understanding and image classification tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
