ValueDCG: Measuring Comprehensive Human Value Understanding Ability of Language Models
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院通用人工智能国家重点实验室);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)
Zhaowei Zhang, Fengshuo Bai, Jun Gao, Yaodong Yang
Personal values are a crucial factor behind human decision-making. Considering that Large Language Models (LLMs) have been shown to impact human decisions significantly, it is essential to make sure they accurately understand human values to ensure their safety. However, evaluating their grasp of these values is complex due to the value's intricate and adaptable nature. We argue that truly understanding values in LLMs requires considering both "know what" and "know why". To this end, we present a comprehensive evaluation metric, ValueDCG (Value Discriminator-Critique Gap), to quantitatively assess the two aspects with an engineering implementation. We assess four representative LLMs and provide compelling evidence that the growth rates of LLM's "know what" and "know why" capabilities do not align with increases in parameter numbers, resulting in a decline in the models' capacity to understand human values as larger amounts of parameters. This may further suggest that LLMs might craft plausible explanations based on the provided context without truly understanding their inherent value, indicating potential risks.
A global analysis of metrics used for measuring performance in natural language processing
Institute of Artificial Intelligence(人工智能研究所);Center for Medical Statistics, Informatics, and Intelligent Systems(医学统计、信息学与智能系统中心);Medical University of Vienna(维也纳医科大学)
测量自然语言处理模型的性能是一项挑战。传统上使用的指标,如BLEU和ROUGE,最初是为机器翻译和摘要设计的,已经被证明与人类判断的相关性低,并且缺乏可转移到其他任务和语言。在过去的15年里,已经提出了广泛的替代指标。然而,目前还不清楚这在多大程度上影响了NLP基准测试工作。在这里,我们提供了第一个大规模的横截面分析的指标用于衡量性能的自然语言处理。我们对来自开放资源库“Papers with Code”的3500多个机器学习模型性能结果进行了策划、映射和系统化,以实现全球和全面的分析。我们的研究结果表明,目前使用的大多数自然语言处理指标都具有可能导致模型性能反映不足的属性。此外,我们发现,指标报告中的模糊性和不一致性可能导致难以解释和比较模型性能,损害NLP研究的透明度和可重复性。
ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatability
IRT Saint Exupéry(圣埃克苏佩里技术研究所);IRIT(图卢兹计算机科学研究所);Université Paul-Sabatier(保罗-萨巴蒂埃大学);Google Research(谷歌研究院);CerCo, CNRS, Université de Toulouse(法国国家科学研究中心, 图卢兹大学);ANITI, Université de Toulouse(图卢兹大学)
HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models
The University of Sheffield(谢菲尔德大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Beijing Academy of Artificial Intelligence(北京人工智能研究院);University of Michigan Ann Arbor(密歇根大学安娜堡分校)
AIMeter: Measuring, Analyzing, and Visualizing Energy and Carbon Footprint of AI Workloads
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));National University of Defense Technology(国防科技大学);University of Victoria(维多利亚大学)
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screening
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Duke University(杜克大学);Arizona State University(亚利桑那州立大学);hireEZ;University of California, Berkeley(加州大学伯克利分校)
RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian
University POLITEHNICA of Bucharest(布加勒斯特理工大学);University of Bucharest, HTL Research Center(布加勒斯特大学HTL研究中心);PRHLT Research Center, Universitat Politècnica de València(瓦伦西亚理工大学PRHLT研究中心);ValgrAI - Valencian Graduate School and Research Network of Artificial Intelligence(ValgrAI - 瓦伦西亚人工智能研究生院与研究网络)