Litmus (Re)Agent：一种用于多语言模型预测评估的基准和代理系统
Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models
AI总结：
本文提出Litmus (Re)Agent代理系统，通过结构化代理推理方法，在缺乏直接证据的情况下评估多语言模型性能，尤其在转移密集场景中表现突出。
摘要：
我们研究了预测多语言评估：估计模型在目标语言任务上的表现，当直接基准结果缺失时。此问题在多语言部署中普遍存在，评估覆盖稀疏且出版证据在语言、任务和模型家族间不均。我们引入了一个包含1500个问题的受控基准，涵盖六个任务和五个证据场景。该基准将可访问的证据与真实结果分离，使系统能够推断缺失结果。我们还提出了Litmus (Re)Agent，一种通过DAG调度的代理系统，将查询分解为假设，检索证据，并通过特征感知聚合合成预测。在六个系统中，Litmus (Re)Agent实现了最佳整体性能，尤其在直接证据弱或缺失的转移密集场景中收益最大。这些结果表明，结构化代理推理是处理不完整证据下多语言性能估计的有前景方法。
英文摘要：
We study predictive multilingual evaluation: estimating how well a model will perform on a task in a target language when direct benchmark results are missing. This problem is common in multilingual deployment, where evaluation coverage is sparse and published evidence is uneven across languages, tasks, and model families. We introduce a controlled benchmark of 1,500 questions spanning six tasks and five evidence scenarios. The benchmark separates accessible evidence from ground truth, enabling evaluation of systems that must infer missing results from incomplete literature evidence. We also present Litmus (Re)Agent, a DAG-orchestrated agentic system that decomposes queries into hypotheses, retrieves evidence, and synthesises predictions through feature-aware aggregation. Across six systems, Litmus (Re)Agent achieves the best overall performance, with the largest gains in transfer-heavy scenarios where direct evidence is weak or absent. These results show that structured agentic reasoning is a promising approach to multilingual performance estimation under incomplete evidence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
