Large language models (LLMs) such as ChatGPT, GPT-4, Claude-3, and Llama are being integrated across a variety of industries. Despite this rapid proliferation, experts are calling for caution in the interpretation and adoption of LLMs, owing to numerous associated ethical concerns. Research has also uncovered shortcomings in LLMs' reasoning and logical abilities, raising questions on the potential of LLMs as evaluation tools. In this paper, we investigate LLMs' self-evaluation capabilities on a novel proverb reasoning task. We introduce a novel proverb database consisting of 300 proverb pairs that are similar in intent but different in wordings, across topics spanning gender, wisdom, and society. We propose tests to evaluate textual consistencies as well as numerical consistencies across similar proverbs, and demonstrate the effectiveness of our method and dataset in identifying failures in LLMs' self-evaluation which in turn can highlight issues related to gender stereotypes and lack of cultural understanding in LLMs.
Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generation
Tehran Institute for Advanced Studies, Khatam University, Iran（德黑兰高级研究所，卡塔姆大学，伊朗）；School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran（电气与计算机工程学院，工程学院，德黑兰大学，伊朗）
MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs
School of Electrical and Computer Engineering, College of Engineering, University of Tehran（德黑兰大学电气与计算机工程学院，工程学院）；Tehran Institute for Advanced Studies, Khatam University（德黑兰高级研究 institute，Khatam 大学）
FFE-Hallu:Hallucinations in Fixed Figurative Expressions:Benchmark of Idioms and Proverbs in the Persian Language
Tehran Institute for Advanced Studies, Khatam University, Iran（德黑兰高级研究 institute，Khatam 大学，伊朗）；School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran（电气与计算机工程学院，工程学院，德黑兰大学，德黑兰，伊朗）
From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher
Department of Computer Science, King Khalid University（国王 Khalid 大学计算机科学系）；Department of Computer Science and Artificial Intelligence, Umm Al-Qura University（乌姆·阿勒·库拉大学计算机科学与人工智能系）；Imam Abdulrahman Bin Faisal University（伊玛目阿卜杜勒拉赫曼·本·法伊萨尔大学）
AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification
Shanghai AI Laboratory（上海人工智能实验室）；MMLab, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Shanghai Jiao Tong University（上海交通大学）；Great Bay University（大湾区大学）
MechMath: Sorrifier-Driven Formal Decomposition Workflow for Automated Theorem Proving
Academy of Mathematics and Systems Science, CAS（数学与系统科学研究院，中国科学院）；School of Advanced Interdisciplinary Sciences, UCAS（交叉科学学院，中国科学院大学）；School of Mathematical Science, UCAS（数学科学学院，中国科学院大学）
StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs
Southern University of Science and Technology（南方科技大学）；Alibaba Group（阿里巴巴集团）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Hong Kong（香港大学）
Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs
Department of Informatics, King's College London, UK（伦敦国王学院信息学院）；Department of Mathematics, Imperial College London, UK（伦敦帝国学院数学系）；Department of Statistics, University of Warwick, UK（沃里克大学统计系）
Does Generative AI speak Nigerian-Pidgin?: Issues about Representativeness and Bias for Multilingualism in LLMs
Mila - Quebec AI Institute McGill University, Canada（魁北克AI研究所麦吉尔大学，加拿大）；LT3, IDLab, Universiteit Gent（LT3，IDLab，根特大学）；Bloomberg（彭博）；University of Trento（特伦托大学）
Raman Lab, Malaviya National Institute of Technology（拉曼实验室，马拉维亚国家理工学院）；Inferigence Quotient Private Limited（Inferigence Quotient私人有限公司）；Language Technologies Research Center, KCIS, IIIT Hyderabad（海德拉巴国际信息技术学院KCIS语言技术研究中心）；Suresh Gyan Vihar University（苏雷什·格扬·维哈尔大学）
