Large Language Model (LLM) evaluation is currently one of the most important areas of research, with existing benchmarks proving to be insufficient and not completely representative of LLMs' various capabilities. We present a curated collection of challenging statements on sensitive topics for LLM benchmarking called TruthEval. These statements were curated by hand and contain known truth values. The categories were chosen to distinguish LLMs' abilities from their stochastic nature. We perform some initial analyses using this dataset and find several instances of LLMs failing in simple tasks showing their inability to understand simple questions.
Truth Knows No Language: Evaluating Truthfulness Beyond English
HiTZ Center - Ixa, University of the Basque Country, UPV/EHU（希茨中心-ixa，巴斯克大学，UPV/EHU）；Elhuyar（埃尔胡亚尔）；Centro de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela（智能技术研究中心（CiTIUS），圣地亚哥-德孔波斯特拉大学）；Departament de Traducció i Ciències del Llenguatge, Universitat Pompeu Fabra（翻译与语言科学部门，庞培法布拉大学）
How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence
University of California, Los Angeles（加州大学洛杉矶分校）；University of Alberta（阿尔伯塔大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Harvard University（哈佛大学）
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；University of Waterloo（滑铁卢大学）
By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction c…
The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
Beijing University of Posts and Telecommunications（北京邮电大学）；Jinan University（暨南大学）；Beihang University（北京航空航天大学）；China Academy of Information and Communications Technology（中国信息通信研究院）
PolBiX: Detecting LLMs' Political Bias in Fact-Checking through X-phemisms
Quality & Usability Lab, Technische Universität Berlin（柏林技术大学质量与可用性实验室）；German Research Center for Artificial Intelligence (DFKI), Berlin（德国人工智能研究中心（DFKI））；Department of Media and Communication, Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学媒体与传播系）
Generalization of Fine-Tuned Uncertainty Communication and Metacognition in Large Language Models
Department of Cognitive Sciences, University of California, Irvine, United States（认知科学系，加州大学伊文斯顿分校，美国）；Department of Computer Science, University of California, Irvine, United States（计算机科学系，加州大学伊文斯顿分校，美国）
Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Meituan（美团）
Transparent NLP: Using RAG and LLM Alignment for Privacy Q&A
Fraunhofer Institute for Integrated Circuits (IIS)（弗劳恩霍夫集成电路研究所）；Ruhr University Bochum（波鸿鲁尔大学）；Research Center Trustworthy Data Science and Security（可信数据科学与安全研究中心）；FIZ Karlsruhe - Leibniz Institute for Information Infrastructure（卡尔斯鲁厄FIZ - 莱布尼茨信息基础设施研究所）；International Audio Laboratories Erlangen（埃尔朗根国际音频实验室）
AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic
Center of Research Excellence in AI and Data Science（人工智能与数据科学卓越研究中心）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；ASAS AI（ASAS人工智能公司）；University College London（伦敦大学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Alexandria University（亚历山大大学）
RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention
Drexel University（德雷塞尔大学）；University of Electronic Science and Technology of China（电子科技大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；LLNL（劳伦斯利弗莫尔国家实验室）；Lehigh University（莱斯大学）
