arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18776cs.CLcs.AI
收藏

阿拉伯数字阅读基准:评估大型语言模型在阿拉伯数字阅读任务中的表现

ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models

Imam Mohammad Ibn Saud Islamic University Saudi Arabia(伊玛目穆罕默德·本·沙特伊斯兰大学)

Anas Alhumud, Abdulaziz Alhammadi, Muhammad Badruddin Khan

AI总结:

阿拉伯数字基准评估大型语言模型在阿拉伯数字阅读任务中的表现,揭示不同提示策略和模型在准确率上的显著差异,指出结构化输出生成的挑战。

摘要:

我们提出了阿拉伯数字基准(ArabicNumBench),这是一个全面的基准,用于评估大型语言模型在阿拉伯数字阅读任务中的表现,涵盖东部阿拉伯-印度数字(0-9在阿拉伯文中)和西方阿拉伯数字(0-9)。我们使用四种提示策略(零样本、零样本CoT、少样本、少样本CoT)在210个数字阅读任务上评估了来自10家供应商的71个模型,这些任务覆盖了六个情境类别:纯数字、地址、日期、数量和价格。我们的评估包含59,010个单独的测试案例,并追踪提取方法以测量结构化输出生成。评估结果显示了显著的表现差异,模型和策略的准确率范围从14.29%到99.05%。少样本链式思维提示方法的准确率比零样本方法高2.8倍(80.06% vs 28.76%)。一个显著发现是:达到精英准确率(98-99%)的模型往往产生主要的非结构化输出,大多数响应缺乏阿拉伯CoT标记。只有6个模型在所有测试案例中一致生成结构化输出,而大多数模型即使在高数值准确性下仍需要回退提取方法。对281个模型-策略组合的全面评估表明,数值准确性和遵循指示是不同的能力,为阿拉伯数字理解建立了基准,并为生产阿拉伯语NLP系统中的模型选择提供了可行的指导。

英文摘要:

We present ArabicNumBench, a comprehensive benchmark for evaluating large language models on Arabic number reading tasks across Eastern Arabic-Indic numerals (0-9 in Arabic script) and Western Arabic numerals (0-9). We evaluate 71 models from 10 providers using four prompting strategies (zero-shot, zero-shot CoT, few-shot, few-shot CoT) on 210 number reading tasks spanning six contextual categories: pure numerals, addresses, dates, quantities, and prices. Our evaluation comprises 59,010 individual test cases and tracks extraction methods to measure structured output generation. Evaluation reveals substantial performance variation, with accuracy ranging from 14.29\% to 99.05\% across models and strategies. Few-shot Chain-of-Thought prompting achieves 2.8x higher accuracy than zero-shot approaches (80.06\% vs 28.76\%). A striking finding emerges: models achieving elite accuracy (98-99\%) often produce predominantly unstructured output, with most responses lacking Arabic CoT markers. Only 6 models consistently generate structured output across all test cases, while the majority require fallback extraction methods despite high numerical accuracy. Comprehensive evaluation of 281 model-strategy combinations demonstrates that numerical accuracy and instruction-following represent distinct capabilities, establishing baselines for Arabic number comprehension and providing actionable guidance for model selection in production Arabic NLP systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑