Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves. We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution. ReFeR-Lite is $\sim7.7\times$ more efficient while being comparably accurate to ReFeR-Turbo. We make code, data and PIP package publicly available. See this PIP URL https://pypi.org/project/refer-agents/ and this Git URL https://github.com/yaswanth-iitkgp/ReFeR_Code .
Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuning
College of Computer Science and Technology of Zhejiang University(浙江大学计算机科学与技术学院);Polytechnic Institute of Zhejiang University(浙江大学Polytechnic学院);Om AI Research(Om AI研究机构);Binjiang Research Institute of Zhejiang University(浙江大学滨江研究机构);School of Software Engineering of Zhejiang University(浙江大学软件工程学院);School of Mathematical Sciences of Zhejiang University(浙江大学数学科学学院);China Academy of Space Technology(中国航天科技研究院);University of Bristol(布里斯托大学)
Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimization
School of Medicine, Stanford University(斯坦福大学医学院);Data Science Institute, University of Chicago(芝加哥大学数据科学研究所);Pritzker School of Molecular Engineering, University of Chicago(芝加哥大学普利兹克分子工程学院);Department of Computer Science, University of Chicago(芝加哥大学计算机科学系);Argonne National Laboratory(阿贡国家实验室)
Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);The Chinese University of Hong Kong(香港中文大学);AIWaves Inc.(AIWaves公司);Microsoft Research Asia, China(微软亚洲研究院);Microsoft(微软公司)