Contemporary vision-language models (VLMs) perform well on existing multimodal reasoning benchmarks (78-85\% accuracy on MMMU, MathVista). Yet, these results fail to sufficiently distinguish true scientific reasoning articulation capabilities from pattern-matching. To address this gap, we introduce \textbf{mmJEE-Eval}, a multimodal bilingual (English and Hindi) benchmark comprising 1,460 questions from India's JEE Advanced examination (2019-2025) spanning pre-college Physics, Chemistry, and Mathematics domains. Our evaluation of 17 state-of-the-art models reveals that while frontier VLMs (GPT-5, Gemini 2.5 Pro/Flash) achieve 77-84\% accuracy on held-out 2025 questions, open-source models plateau at 37-45\% despite scaling to 400B parameters, a significant difference not observed on existing benchmarks. While closed frontiers from Google and OpenAI show high problem-solving accuracies (up to 100\% pass@3 scores), they fully collapse when the reasoning load is increased meta-cognitively (GPT-5 fixes just 5.2\% errors). Systematic ablations show mmJEE-Eval's difficulty stems from complexity and reasoning depth rather than memorization. Effectively, our benchmark segregates superior training and reasoning methodologies where alternatives fail. We publicly release our code and data: https://mmjee-eval.github.io
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
University of British Columbia(不列颠哥伦比亚大学);Zuse School(Zuse学校);Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所);Hamad Bin Khalifa University(哈马德·本·哈利法大学)
A Novel Bilingual Word Embedding Method for Lexical Translation Using Bilingual Sense Clique
Shanghai Jiao Tong University(上海交通大学);Centre National de la Recherche Scientifique(法国国家科学研究中心);National Institute of Information and Communications Technology(信息通信研究机构)
现有的双语词嵌入方法大多只考虑了浅上下文或简单的共现信息。本文提出了一种潜在双语语义单元(Bilingual Sense Clique,BSC),它是由双语语料库上基于逐点互信息的图的最大完全子图导出的。这样,我们平等地对待源和目标词,并且不再需要在大多数现有作品中必须使用的单独的双语投影处理。几种降维方法进行评估,总结BSC的字的关系。在双语词典翻译任务上对所提出的方法进行了评估,实验结果表明,双语意义嵌入方法优于现有的双语词嵌入方法。
Qorgau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contexts
Mohamed bin Zayed University of Artificial Intelligence(莫扎伊德大学人工智能学院);Monash University(莫纳什大学);University of Copenhagen(哥本哈根大学);The University of Melbourne(墨尔本大学);LibrAI(LibrAI公司);Institute of Foundation Models(基础模型研究所)
Dólares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English
The Fin AI;University of Manchester(曼彻斯特大学);School of Computer Science Sichuan University(四川大学计算机学院);School of Computer Science Wuhan University(武汉大学计算机学院);University of Florida(佛罗里达大学);Columbia University(哥伦比亚大学)
针对西班牙语金融NLP研究不足的问题,提出首个双语框架Toisón de Oro,构建指令数据集与评估基准,并推出FinMA-ES模型,在西班牙语金融任务中超越GPT-4等SOTA模型。
RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);China Academy of Information and Communications Technology(中国信息通信研究院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models
School of Future Technology, Dalian University of Technology(大连理工大学未来技术学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Public Administration and Policy, Dalian University of Technology(大连理工大学公共管理与政策学院)
Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models
IDEA Research(IDEA研究机构);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);Nanjing University(南京大学);South China Normal University(华南师范大学);DataArcTech Ltd.(DataArcTech有限公司)
Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer
Joint Laboratory of HIT and iFLYTEK (HFL), iFLYTEK Research(哈尔滨工业大学与科大讯飞联合实验室(HFL),科大讯飞研究院);Harbin Institute of Technology(哈尔滨工业大学);iFLYTEK AI Research (Hebei)(科大讯飞人工智能研究院(河北));State Key Laboratory of Cognitive Intelligence, iFLYTEK Research(认知智能国家重点实验室,科大讯飞研究院)
From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages
OpenBabylon;ARG-Tech, University of Dundee(邓迪大学ARG-Tech公司);Georg-August Universität Göttingen(哥廷根大学);Doha Institute for Graduate Studies(多哈研究生学院);Arab Center for Research and Policy Studies(阿拉伯研究与政策研究中心);Google(谷歌);PolyAgent