Just Rank: Rethinking Evaluation with Word and Sentence Similarities
National University of Singapore（新加坡国立大学）；University of Southern California（南加州大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Kriston AI（科通智能）
Word and sentence embeddings are useful feature representations in natural language processing. However, intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. Word and sentence similarity tasks have become the de facto evaluation method. It leads models to overfit to such evaluations, negatively impacting embedding models' development. This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations. Further, we propose a new intrinsic evaluation method called EvalRank, which shows a much stronger correlation with downstream tasks. Extensive experiments are conducted based on 60+ models and popular datasets to certify our judgments. Finally, the practical evaluation toolkit is released for future benchmarking purposes.
Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark
King Abdullah University of Science and Technology（国王阿卜杜勒·阿齐兹科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；National Institutes of Health（美国国家卫生研究院）；University of Notre Dame（诺特难大学）；Shandong University（山东大学）
Rethinking the Understanding Ability across LLMs through Mutual Information
Department of Industrial and Systems Engineering（工业与系统工程系）；University of Houston（休斯敦大学）；Bakar Computational Health Sciences Institute（巴卡计算健康科学研究所）；UCSF
Simultaneous machine translation (SimulMT) models start translation before the end of the source sentence, making the translation monotonically aligned with the source sentence. However, the general full-sentence translation test set is acquired by offline translation of the entire source sentence, which is not designed for SimulMT evaluation, making us rethink whether this will underestimate the performance of SimulMT models. In this paper, we manually annotate a monotonic test set based on the MuST-C English-Chinese test set, denoted as SiMuST-C. Our human evaluation confirms the acceptabili…
Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation
ADAPT Centre, Munster Technological University（ADAPT中心，芒斯特理工大学）；Nimbus Research Centre, Munster Technological University（宁巴斯研究中心，芒斯特理工大学）；EEECS, Queen’s University Belfast（电子、电气与计算机工程学院，贝尔法斯特女王大学）
Rethinking Text Attribute Transfer: A Lexical Analysis
Columbia University（哥伦比亚大学）；Bytedance AI Lab（字节跳动人工智能实验室）
文本属性迁移是指对某些语言属性（如情感、风格、作者等）的修改。并将它们从一种类型转换为另一种类型。在本文中，我们的目的是分析和解释在转移过程中发生了什么变化。我们首先观察到，在许多现有的模型和数据集中，句子中的某些单词在确定句子属性类方面发挥着重要作用。这些单词称为\textit{the Pivot Words}。基于这些主元词，我们提出了一个词汇分析框架\textit{the Pivot Analysis}，定量分析这些词在文本属性分类和迁移中的作用。我们将此框架应用于现有的数据集和模型，并表明：（1）主元词是句子属性分类的强特征;（2）要改变句子的属性，许多数据集只需要改变某些主元词;（3）因此，许多迁移模型只执行词汇级别的修改，而不改变更高级别的句子结构。我们的工作提供了一个深入的了解语言属性转移，并进一步确定了未来的要求和挑战，这项任务\footnote{Our code can be found at https://github.com/FranxYao/pivot_analysis}。
Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Cyber Science and Engineering, Huazhong University of Science and Technology（华中科技大学网络空间安全学院）
在本文中，我们注意到一个通常被忽视的问题，即，\textit{similarity should be determined from different perspectives}。为了探索这个问题，我们发布了一个多视角文本相似性（MPTS）数据集，其中句子相似性从12个角度进行标记。在此基础上，我们对一些著名的文本匹配模型进行了改进，并进行了一系列的实验分析。最后，本文得出了几点结论和基准模型，为后续研究奠定了基础。数据集和代码可在Github \footnote { \url{https://github.com/autoliuweijie/MPTS}
Queen Mary University of London（伦敦玛丽女王大学）；Taif University（塔伊夫大学）
An important component of an automated fact-checking system is the claim check-worthiness detection system, which ranks sentences by prioritising them based on their need to be checked. Despite a body of research tackling the task, previous research has overlooked the challenging nature of identifying check-worthy claims across different topics. In this paper, we assess and quantify the challenge of detecting check-worthy claims for new, unseen topics. After highlighting the problem, we propose the AraCWA model to mitigate the performance deterioration when detecting check-worthy claims across…
DefSent+: Improving sentence embeddings of language models by projecting definition sentences into a quasi-isotropic or isotropic vector space of unlimited dictionary entries
Variants of the BERT architecture specialised for producing full-sentence representations often achieve better performance on downstream tasks than sentence embeddings extracted from vanilla BERT. However, there is still little understanding of what properties of inputs determine the properties of such representations. In this study, we construct several sets of sentences with pre-defined lexical and syntactic structures and show that SOTA sentence transformers have a strong nominal-participant-set bias: cosine similarities between pairs of sentences are more strongly determined by the overlap…
Structural-Aware Sentence Similarity with Recursive Optimal Transport
Research Institute of Information Technology（信息技术研究所）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank
Peking University（北京大学）；Meituan（美团）；Meta AI；National Key Laboratory of General Artificial Intelligence（通用人工智能全国重点实验室）；BIGAI（北京通用人工智能研究院）；Renmin University of China（中国人民大学）
Guiding Neural Machine Translation with Retrieved Translation Pieces
National Institute of Information and Communications Technology（国立信息通信技术研究所）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Carnegie Mellon University（卡内基梅隆大学）
May I Check Again? -- A simple but efficient way to generate and use contextual dictionaries for Named Entity Recognition. Application to French Legal Texts
Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；NTT Communication Science Laboratories（NTT通信科学实验室）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所革新智能综合研究中心）
