arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09757cs.LGcs.CLcs.CY
收藏

用于提升大语言模型上下文学习公平性的策略性示范样本选择

Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning

University of Bristol(布里斯托尔大学);New Jersey Institute of Technology(新泽西理工学院)

Jingyu Hu, Weiru Liu, Mengnan Du

AI总结:

本研究探究上下文学习提示中示范样本对大语言模型公平性的影响,提出结合聚类与进化策略的样本筛选方法,在保障预测准确率的同时有效提升模型公平性。

摘要:

近期研究凸显了利用上下文学习(ICL)引导大语言模型(LLMs)处理表格数据的有效性,鉴于此类数据的结构化特性,这是一项颇具挑战性的任务。尽管性能方面有所进展,但这些方法在公平性层面的影响仍未得到充分理解。本研究探究了ICL提示中不同示范样本如何影响LLMs的公平性结果。研究发现,在提示中刻意纳入少数群体样本可在不牺牲预测准确率的前提下显著提升公平性。进一步实验表明,示范样本中少数群体与多数群体样本的比例会影响公平性与预测准确率之间的权衡。基于这些发现,我们提出了一种缓解技术,该技术采用聚类和进化策略从训练数据中筛选出兼具多样性与代表性的样本集。该方法旨在同时提升ICL应用中的预测性能与公平性。实验结果证实,我们提出的方法在多项指标上大幅提升了公平性,彰显了其在现实场景中的有效性。

英文摘要:

Recent studies highlight the effectiveness of using in-context learning (ICL) to steer large language models (LLMs) in processing tabular data, a challenging task given the structured nature of such data. Despite advancements in performance, the fairness implications of these methods are less understood. This study investigates how varying demonstrations within ICL prompts influence the fairness outcomes of LLMs. Our findings reveal that deliberately including minority group samples in prompts significantly boosts fairness without sacrificing predictive accuracy. Further experiments demonstrate that the proportion of minority to majority samples in demonstrations affects the trade-off between fairness and prediction accuracy. Based on these insights, we introduce a mitigation technique that employs clustering and evolutionary strategies to curate a diverse and representative sample set from the training data. This approach aims to enhance both predictive performance and fairness in ICL applications. Experimental results validate that our proposed method dramatically improves fairness across various metrics, showing its efficacy in real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑