We address the task of evaluating image description generation systems. We propose a novel image-aware metric for this task: VIFIDEL. It estimates the faithfulness of a generated caption with respect to the content of the actual image, based on the semantic similarity between labels of objects depicted in images and words in the description. The metric is also able to take into account the relative importance of objects mentioned in human reference descriptions during evaluation. Even if these human reference descriptions are not available, VIFIDEL can still reliably evaluate system descriptions. The metric achieves high correlation with human judgments on two well-known datasets and is competitive with metrics that depend on human references
InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
Fudan University(复旦大学);Carnegie Mellon University(卡内基梅隆大学);The Chinese University of Hong Kong(香港中文大学);Renmin University of China(中国人民大学);Beijing University of Technology(北京工业大学);Boston University(波士顿大学);RhineAI(莱茵AI公司);SenseTime(商汤科技)
CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(广东省新型安全智能技术重点实验室,网络空间安全研究院,计算机科学与技术学院,哈尔滨工业大学(深圳))
How Real Are Synthetic Therapy Conversations? Evaluating Fidelity in Prolonged Exposure Dialogues
College of Information Sciences and Technology, Penn State University(宾夕法尼亚州立大学信息科学与技术学院);Department of Psychiatry and Behavioral Sciences, Emory University(埃默里大学精神病学与行为科学系);School of Interactive Computing, Georgia Tech(佐治亚理工学院交互计算学院);School of Psychology, Georgia Tech(佐治亚理工学院心理学学院)
TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots
Department of Computer Science, Stanford University, Stanford, CA, USA(计算机科学系,斯坦福大学,斯坦福,加州,美国);Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA(精神病学与行为科学系,斯坦福大学,斯坦福,加州,美国);Department of Biomedical Data Science, Stanford University, Stanford, CA, USA(生物医学数据科学系,斯坦福大学,斯坦福,加州,美国);University of California, Berkeley, Berkeley, CA, USA(加州大学伯克利分校,伯克利,加州,美国);The University of Hong Kong, Hong Kong(香港大学,香港)
Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations
Technical University of Munich (TUM)(慕尼黑技术大学);TUM University Hospital(慕尼黑技术大学医院);German Heart Center TUM University Hospital(慕尼黑技术大学医院德国心脏中心);Department of Radiology(放射科);Klinikum rechts der Isar TUM University Hospital(慕尼黑技术大学医院右岸诊所);Uniklinik RWTH Aachen(亚琛工业大学医院);HOPPR IL USA(HOPPR美国);University of Oxford(牛津大学);Imperial College London(伦敦帝国学院)
Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation
School of Foreign Languages, Shanghai Jiao Tong University(上海交通大学外国语学院);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);Institute of Corpus Studies and Applications, Shanghai International Studies University(上海外国语大学语料库研究与应用研究所)