Multimodal machine translation (MMT) systems have been shown to outperform their text-only neural machine translation (NMT) counterparts when visual context is available. However, recent studies have also shown that the performance of MMT models is only marginally impacted when the associated image is replaced with an unrelated image or noise, which suggests that the visual context might not be exploited by the model at all. We hypothesize that this might be caused by the nature of the commonly used evaluation benchmark, also known as Multi30K, where the translations of image captions were prepared without actually showing the images to human translators. In this paper, we present a qualitative study that examines the role of datasets in stimulating the leverage of visual modality and we propose methods to highlight the importance of visual signals in the datasets which demonstrate improvements in reliance of models on the source images. Our findings suggest the research on effective MMT architectures is currently impaired by the lack of suitable datasets and careful consideration must be taken in creation of future MMT datasets, for which we also provide useful insights.
Machine translation is research based area where evaluation is very important phenomenon for checking the quality of MT output. The work is based on the evaluation of English to Urdu Machine translation. In this research work we have evaluated the translation quality of Urdu language which has been translated by using different Machine Translation systems like Google, Babylon and Ijunoon. The evaluation process is done by using two approaches - Human evaluation and Automatic evaluation. We have worked for both the approaches where in human evaluation emphasis is given to scales and parameters …
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA（卡内基梅隆大学机器学习系）；Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA（加州大学圣地亚哥分校计算机科学与工程系）；Department of Computer Science, University of Virginia, Charlottesville, VA, USA（弗吉尼亚大学计算机科学系）；Martian Research, San Francisco, California, USA（火星研究公司）；Apart Research, San Francisco, California, USA（Apart研究公司）
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Common Sense Advisory (CSA Research)（常识咨询公司（CSA研究））
We present an alternative method of evaluating Quality Estimation systems, which is based on a linguistically-motivated Test Suite. We create a test-set consisting of 14 linguistic error categories and we gather for each of them a set of samples with both correct and erroneous translations. Then, we measure the performance of 5 Quality Estimation systems by checking their ability to distinguish between the correct and the erroneous translations. The detailed results are much more informative about the ability of each system. The fact that different Quality Estimation systems perform differentl…
The Hong Kong University of Science and Technology（香港科技大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huazhong University of Science and Technology（华中科技大学）
Machine translation has wide applications in daily life. In mission-critical applications such as translating official documents, incorrect translation can have unpleasant or sometimes catastrophic consequences. This motivates recent research on testing methodologies for machine translation systems. Existing methodologies mostly rely on metamorphic relations designed at the textual level (e.g., Levenshtein distance) or syntactic level (e.g., the distance between grammar structures) to determine the correctness of translation results. However, these metamorphic relations do not consider whether…
Centre for Translation Studies（翻译研究中心）；University of Surrey（萨里大学）；RGCL（语言计算研究组）；University of Wolverhampton（伍尔弗汉普顿大学）
Although emotions are universal concepts, transferring the different shades of emotion from one language to another may not always be straightforward for human translators, let alone for machine translation systems. Moreover, the cognitive states are established by verbal explanations of experience which is shaped by both the verbal and cultural contexts. There are a number of verbal contexts where expression of emotions constitutes the pivotal component of the message. This is particularly true for User-Generated Content (UGC) which can be in the form of a review of a product or a service, a …
Speech translation has recently become an increasingly popular topic of research, partly due to the development of benchmark datasets. Nevertheless, current datasets cover a limited number of languages. With the aim to foster research in massive multilingual speech translation and speech translation for low resource language pairs, we release CoVoST 2, a large-scale multilingual speech translation corpus covering translations from 21 languages into English and from English into 15 languages. This represents the largest open dataset available to date from total volume and language coverage pers…
Fair Fact-Checking: Closing the Cross-Lingual Gap in LLM Factual Judgement with RoSh
Shifa Tameer-e-Millat University（希法塔米尔米拉特大学）；BRAINS, Brandenburg Research Center for Applied Intelligent Systems（勃兰登堡应用智能系统研究中心）；Max Planck Institute for Security and Privacy（马克斯·普朗克安全与隐私研究所）；University of Cambridge（剑桥大学）；The Pennsylvania State University（宾夕法尼亚州立大学）；GISMA University of Applied Sciences（GISMA应用科学大学）
Lost in translation: using global fact-checks to measure multilingual misinformation prevalence, spread, and evolution
Department of Mathematical Modeling and Machine Learning, University of Zurich, Switzerland（苏黎世大学数学建模与机器学习系）；Digital Society Initiative, University of Zurich, Switzerland（苏黎世大学数字社会倡议）；Oxford Internet Institute, University of Oxford, United Kingdom（牛津大学网络研究所）；Meedan, San Francisco, United States（梅丹公司，旧金山，美国）
Pipeline and Dataset Generation for Automated Fact-checking in Almost Any Language
Artificial Intelligence Center, Faculty of Electrical Engineering, Czech Technical University in Prague（捷克技术大学布拉格校区电气工程学院人工智能中心）；Department of Journalism, Faculty of Social Sciences, Charles University（查理大学社会科学学院新闻学系）
MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage
Mohamed bin Zayed University of Artificial Intelligence, UAE（马尔代夫布扎伊德人工智能大学，阿联酋）；National Institute of Technology, Silchar（西尔CHAR国家理工学院）；Birmingham City University, UK（伯明翰城市大学，英国）
