With the onset of large language models (LLMs), the performance of artificial intelligence (AI) models is becoming increasingly multi-dimensional. Accordingly, there have been several large, multi-dimensional evaluation frameworks put forward to evaluate LLMs. Though these frameworks are much more realistic than previous attempts which only used a single score like accuracy, multi-dimensional evaluations can complicate decision-making since there is no obvious way to select an optimal model. This work introduces preference sampling, a method to extract a scalar trustworthiness score from multi-dimensional evaluation results by considering the many characteristics of model performance which users value. We show that preference sampling improves upon alternate aggregation methods by using multi-dimensional trustworthiness evaluations of LLMs from TrustLLM and DecodingTrust. We find that preference sampling is consistently reductive, fully reducing the set of candidate models 100% of the time whereas Pareto optimality never reduces the set by more than 50%. Likewise, preference sampling is consistently sensitive to user priors-allowing users to specify the relative weighting and confidence of their preferences-whereas averaging scores is intransigent to the users' prior knowledge.
Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control
State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室);Alibaba Cloud(阿里云);Hong Kong Baptist University(香港浸会大学);School of Software Technology, Zhejiang University(浙江大学软件学院);Zhiyuan Research Institute(智源研究院);Fabu Inc.(fabu公司)
SPOGW: a Score-based Preference Optimization method via Group-Wise comparison for workflows
Hangzhou International Innovation Institute and , Beihang University(杭州国际创新研究院和 北航);School of Artificial Intelligence, Beihang University(人工智能学院,北航);Nanyang Technological University(南洋理工大学);University of Leicester(莱斯特大学);China Mobile Communications Company Limited Research Institute(中国移动通信有限公司研究院)
Quantifying and Mitigating Self-Preference Bias of LLM Judges
CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China(复杂实验室、计算机科学与工程学院、电子科技大学、成都、中国)
ChatPlanner: A Large Language Model Framework for Personalized Public Transit Routing
School of Engineering and Materials Science, Queen Mary University of London(伦敦玛丽女王大学工程与材料科学学院);Department of Engineering Science, University of Oxford(牛津大学工程科学系)
Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?
Department of Information Systems and Business Analytics(信息系统与商业分析系);University at Albany - State University of New York(阿尔巴尼大学 - 新 york 状态大学);Santa Clara, CA(圣克拉拉,加州);Princeton Junction, NJ(普林斯顿 junction,新泽西)
Revealing Preference in Popular Music Through Familiarity and Brain Response
据报道,音乐偏好是一个因素,它可以引起内心的音乐情感,需要准确的地面实况数据和音乐治疗的效率。本研究透过熟悉度、响应时间(反应率)及脑电反应(EEG),以探讨音乐偏好的差异。20名受试者在听了两种类型的流行音乐的合唱部分:没有歌词的音乐(旋律)和有歌词的音乐(歌曲)后进行自我评估。\textcolor{red}{We then conduct a music preference classification using a support vector machine, random forest, and k-nearest neighbors with the familiarity scores, the response rates, and EEG as the feature vectors. The statistical analysis and F1-score of EEG are congruent, which is the brain's right side outperformed its left side in classification performance.}最后,这些行为和大脑的研究支持,偏好,熟悉度和反应率可以有助于音乐情感实验的设计,以了解音乐,情感和听众。不仅对音乐行业,生物医学和医疗保健行业也可以利用这个实验来收集患者的数据,以…
FERGI: Automatic Scoring of User Preferences for Text-to-Image Generation from Spontaneous Facial Expression Reaction
Neurosciences Graduate Program, University of California San Diego(加州大学圣地亚哥分校神经科学研究生项目);Department of Computer Science and Engineering, University of California San Diego(加州大学圣地亚哥分校计算机科学与工程系);Department of Cognitive Science, University of California San Diego(加州大学圣地亚哥分校认知科学系);Halıcıoğlu Data Science Institute, University of California San Diego(加州大学圣地亚哥分校Halıcıoğlu数据科学研究所)