Robust, faithful and harm-free pronoun use for individuals is an important goal for language model development as their use increases, but prior work tends to study only one or two of these characteristics at a time. To measure progress towards the combined goal, we introduce the task of pronoun fidelity: given a context introducing a co-referring entity and pronoun, the task is to reuse the correct pronoun later. We present RUFF, a carefully-designed dataset of over 5 million instances to measure robust pronoun fidelity in English, and we evaluate 37 model variants from nine popular families, across architectures (encoder-only, decoder-only and encoder-decoder) and scales (11M-70B parameters). When an individual is introduced with a pronoun, models can mostly faithfully reuse this pronoun in the next sentence, but they are significantly worse with she/her/her, singular they and neopronouns. Moreover, models are easily distracted by non-adversarial sentences discussing other people; even one sentence with a distractor pronoun causes accuracy to drop on average by 34 percentage points. Our results show that pronoun fidelity is not robust, in a simple, naturalistic setting where humans achieve nearly 100% accuracy. We encourage researchers to bridge the gaps we find and to carefully evaluate reasoning in settings where superficial repetition might inflate perceptions of model performance.
Probing Social Identity Bias in Chinese LLMs with Gendered Pronouns and Social Groups
Department of Electronics, Information and Bioengineering, Politecnico di Milano(电子、信息与生物工程系,米兰理工大学);University of Science and Technology of China(中国科学技术大学)
How Much Do LLMs Know About Chinese Zero Pronouns?
Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning(湖北省人工智能与智能学习重点实验室);National Language Resources Monitoring and Research Center for Network Media(网络媒体语言资源监测与研究中心);School of Computer Science, Central China Normal University(中央财经大学计算机学院)
English-Japanese Neural Machine Translation with Encoder-Decoder-Reconstructor
Tokyo Metropolitan University(东京都立大学)
神经机器翻译(NMT)最近在机器翻译领域变得流行。然而,NMT在翻译中存在着重词或漏词的问题。为了解决这个问题,Tu et al.(2017)提出了一种使用反向翻译的NMT编码器-解码器-重建器框架。在这种方法中,他们以与Bahdanau等人(2015)相同的方式选择了最佳的前向翻译模型,然后训练了一个双向翻译模型作为微调。他们的实验表明,它提供了显着提高BLEU分数在汉英翻译任务。我们证实,我们的重新实施也表现出同样的趋势,并重申了问题的重复和遗漏的单词在翻译的英日任务太多。此外,我们通过将其与前向翻译和后向翻译的联合训练模型进行比较来评估预训练的有效性。
Dólares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English
The Fin AI;University of Manchester(曼彻斯特大学);School of Computer Science Sichuan University(四川大学计算机学院);School of Computer Science Wuhan University(武汉大学计算机学院);University of Florida(佛罗里达大学);Columbia University(哥伦比亚大学)
针对西班牙语金融NLP研究不足的问题,提出首个双语框架Toisón de Oro,构建指令数据集与评估基准,并推出FinMA-ES模型,在西班牙语金融任务中超越GPT-4等SOTA模型。
Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction
LMU Munich(慕尼黑大学);Uppsala University(乌普萨拉大学);Qatar Computing Res. Inst.(卡塔尔计算研究所);HBKU(哈马德·本·哈利法大学);University of Helsinki(赫尔辛基大学);LinkedIn(领英);Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会);Idiap Research Institute(伊迪亚普研究所)
Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese
ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages
Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLMs
Erasmus Mundus Master in Language and Communication Technologies(埃拉斯谟 Mundus 语言与沟通技术硕士项目);HiTZ Center - Ixa, University of the Basque Country (UPV/EHU)(巴斯克国家大学(UPV/EHU)HiTZ中心 - Ixa);Cohere(Cohere公司);Saarland University(萨尔兰州大学)
MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning
BRAC University(布拉克大学);United International University(国际联合大学);Qatar Computing Research Institute(卡塔尔计算研究所);Center for Computational & Data Sciences, Independent University, Bangladesh(计算与数据科学中心,独立大学,孟加拉国);Amazon GenAI(亚马逊生成人工智能);University of Virginia(弗吉尼亚大学)
Classifying Dominant Temporal Orientation without Pretrained Text Embeddings: A Novel Morphosyntactic Inventory Vector Approach
Interdisciplinary Center for Innovative Theory and Empirics, Columbia University(哥伦比亚大学创新理论与实证跨学科中心);Department of Sociology, Columbia University(哥伦比亚大学社会学系)
XToM: Exploring the Multilingual Theory of Mind for Large Language Models
HKUST, Hong Kong(香港理工大学);NVIDIA AI Technology Center (NVAITC), USA(NVIDIA AI技术中心(NVAITC));Shanghai Jiao Tong University, China(上海交通大学);Technische Universtität Darmstadt, Germany(德累斯顿技术大学);LMU Munich, Germany(慕尼黑大学);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心)
A Predictive Model for Notional Anaphora in English
Georgetown University(乔治城大学);Department of Linguistics(语言学系)
实词回指是指由于实词的原因而与先行词的语法范畴不一致的代词,如在"the government ...“中复数与单数的一致。他们。由于这种情况是罕见的,并与严格同意案件的证据相冲突(“政府...... it“),它们对共指消解和指称表达式生成都提出了实质性的挑战。本文利用OntoNotes语料库,在迄今为止最大的实证数据基础上,采用集成方法预测英语概念回指。除了国家的最先进的预测准确性,结果表明,理论方法假定在先行词的话语是不够的,在回指的话语位置的情况下,以及全球因素,如体裁,有很强的影响的选择指的表达。
Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Study
Ionian University(爱奥尼亚大学);Laboratory on Digital Libraries and Electronic Publishing(数字图书馆与电子出版实验室);Centre for Literary Translation and Book Policy(文学翻译与图书政策中心)