arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.12877cs.CLcs.AI
收藏

ReFeR:通过模型层级结构提升评估与推理能力

ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

IIT Kharagpur(印度理工学院克勒格布尔校区);University of Waterloo(滑铁卢大学);Microsoft, India(微软印度公司)

Yaswanth Narsupalli, Abhranil Chandra, Sreevatsa Muppirala, Manish Gupta, Pawan Goyal

AI总结:

本研究提出无需调优的两级模型层级框架ReFeR,可评估文本、图像类生成输出并支持推理任务,性能超越现有基准,还推出高效、低成本两个变体并开源相关资源。

摘要:

评估大语言模型、视觉语言模型等生成式模型的输出质量面临显著挑战。传统评估方法要么依赖耗费大量资源的人工评估,要么采用与人类判断相关性较低的自动指标。另一类常用方法是使用深度学习系统,这类系统不仅消耗大量计算资源与时间,还需要海量训练数据。本研究提出一种无需调优的框架ReFeR,该框架利用由大语言模型(LLM)和视觉语言模型(VLM)构成的两级层级结构,可评估文本、图像等各类生成式输出。我们在四类不同的评估任务上对ReFeR框架进行了严格验证,结果显示该框架不仅提升了评估准确率,超越了此前的基准水平,还能生成建设性反馈。值得注意的是,该框架同样适用于推理任务,在四类推理任务上的实验表明其具备更优异的集体推理能力。我们提出了该框架的两个变体:针对性能加速优化的ReFeR-Turbo,以及更具成本效益的ReFeR-Lite。ReFeR-Lite的效率约为ReFeR-Turbo的7.7倍,且准确率与后者相当。我们已公开代码、数据与PIP包,可访问PIP地址https://pypi.org/project/refer-agents/ 与Git地址https://github.com/yaswanth-iitkgp/ReFeR_Code 获取。

英文摘要:

Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves. We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution. ReFeR-Lite is $\sim7.7\times$ more efficient while being comparably accurate to ReFeR-Turbo. We make code, data and PIP package publicly available. See this PIP URL https://pypi.org/project/refer-agents/ and this Git URL https://github.com/yaswanth-iitkgp/ReFeR_Code .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑