arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07891cs.CLcs.LG
收藏

面向文本生成的标签高效模型选择

Label-Efficient Model Selection for Text Generation

IBM Research(IBM研究院);Bar-Ilan University(巴伊兰大学);MIT(麻省理工学院)

Shir Ashury-Tahan, Ariel Gera, Benjamin Sznajder, Leshem Choshen, Liat Ein-Dor, Eyal Shnarch

AI总结:

本文提出DiffUse方法,通过聚类模型输出语义差异嵌入来智能选择实例,从而在保持评估可靠性的同时将文本生成模型选择所需的偏好标注数量最多减少75%。

摘要:

针对特定目标任务进行模型选择可能代价高昂,因为这可能需要对不同模型输出质量进行大量标注。我们提出DiffUse,一种基于偏好标注在候选文本生成模型之间做出明智决策的高效方法。DiffUse减少了所需的标注数量,从而在执行评估时节省宝贵的时间和资源。DiffUse通过对表示模型输出之间语义差异的嵌入进行聚类来智能地选择实例。因此,它能够识别出对偏好决策更具信息量的示例子集。我们的方法与模型无关,可应用于任何文本生成模型,用于在模型、提示和配置之间进行选择。此外,我们提出了一种实用的迭代方法,用于动态确定需要标注多少实例。在涵盖数百个模型对的一系列实验中,我们证明DiffUse能够大幅减少所需的标注数量——最多减少75%——同时保持较高的评估可靠性。

英文摘要:

Model selection for a given target task can be costly, as it may entail extensive annotation of the quality of outputs of different models. We introduce DiffUse, an efficient method to make an informed decision between candidate text generation models based on preference annotations. DiffUse reduces the required amount of annotations, thus saving valuable time and resources in performing evaluation. DiffUse intelligently selects instances by clustering embeddings that represent the semantic differences between model outputs. Thus, it is able to identify a subset of examples that are more informative for preference decisions. Our method is model-agnostic, and can be applied to any text generation model for selecting between models, prompts and configurations. Moreover, we propose a practical iterative approach for dynamically determining how many instances to annotate. In a series of experiments over hundreds of model pairs, we demonstrate that DiffUse can dramatically reduce the required number of annotations -- by up to 75% -- while maintaining high evaluation reliability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑