arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18824cs.SIcs.AI
收藏

UniRank:一种多智能体校准流程,用于从匿名化的引文计量信号中估算大学排名

UniRank: A Multi-Agent Calibration Pipeline for Estimating University Rankings from Anonymized Bibliometric Signals

NotionWave

Pedram Riyazimehr, Seyyed Ehsan Mahmoudi

AI总结:

UniRank通过多智能体校准流程,利用匿名化引文数据估算大学排名,实现零记忆误差和高准确性。

摘要:

我们提出了UniRank,一种多智能体LLM流程,利用仅公开可用的OpenAlex和Semantic Scholar的引文计量数据来估算全球排名系统中的大学位置。该系统采用三阶段架构:(a) 从匿名化的机构指标进行零样本估计,(b) 针对真实排名大学的每系统工具增强校准,(c) 最终综合。关键的是,机构被匿名化——名称、国家、DOI、论文标题和合作国家都被抹去——并且其实际排名在评估期间隐藏于校准工具,防止LLM记忆干扰结果。在Times Higher Education (THE) 世界大学排名(n=352)上,该系统实现了MAE=251.5名位置,中位AE=131.5,PNMAE=12.03%,Spearman ρ=0.769,Kendall τ=0.591,50名命中率=20.7%,100名命中率=39.8%,以及记忆指数为零(所有352所大学中没有精确匹配的零宽度预测)。系统性的正向误差(+190.1名,表明系统始终预测比实际更差的排名)以及从精英层级(MAE=60.5,100名命中率=90.5%)到尾部层级(MAE=328.2,100名命中率=20.8%)的单调性能退化,提供了强有力的证据,证明该流程进行的是真正的分析推理,而非回忆记忆的排名。一个实时演示可在https://unirank.scinito.ai上获得。

英文摘要:

We present UniRank, a multi-agent LLM pipeline that estimates university positions across global ranking systems using only publicly available bibliometric data from OpenAlex and Semantic Scholar. The system employs a three-stage architecture: (a) zero-shot estimation from anonymized institutional metrics, (b) per-system tool-augmented calibration against real ranked universities, and (c) final synthesis. Critically, institutions are anonymized -- names, countries, DOIs, paper titles, and collaboration countries are all redacted -- and their actual ranks are hidden from the calibration tools during evaluation, preventing LLM memorization from confounding results. On the Times Higher Education (THE) World University Rankings ($n=352$), the system achieves MAE = 251.5 rank positions, Median AE = 131.5, PNMAE = 12.03%, Spearman $ρ= 0.769$, Kendall $τ= 0.591$, hit rate @50 = 20.7%, hit rate @100 = 39.8%, and a Memorization Index of exactly zero (no exact-match zero-width predictions among all 352 universities). The systematic positive-signed error (+190.1 positions, indicating the system consistently predicts worse ranks than actual) and monotonic performance degradation from elite tier (MAE = 60.5, hit@100 = 90.5%) to tail tier (MAE = 328.2, hit@100 = 20.8%) provide strong evidence that the pipeline performs genuine analytical reasoning rather than recalling memorized rankings. A live demo is available at https://unirank.scinito.ai .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑