In this paper, we generalize text infilling (e.g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-training objective. SSR provides more fine-grained learning signals for text representations by supervising the model to rewrite imperfect spans to ground truth, and it is more consistent than text infilling with many downstream seq2seq tasks that rewrite a source sentences into a target sentence. Our experiments with T5 models on various seq2seq tasks show that SSR can substantially improve seq2seq pre-training. Moreover, we observe SSR is especially helpful to improve pre-training a small-size seq2seq model with a powerful imperfect span generator, which indicates a new perspective of transferring knowledge from a large model to a smaller model for seq2seq pre-training.
DP-IPI: A Hybrid Differential Privacy Text Rewriting Mechanism for Indirect Personal Identifiers in Clinical Texts
Technical University of Berlin(柏林工业大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心)
Improving Multi-turn Dialogue Modelling with Utterance ReWriter
Tencent Inc(腾讯公司);MPI Informatics(马克斯·普朗克信息学研究所);Saarland Informatics Campus(萨尔大学信息学院);Institute of Software, University of Chinese Academy of Science(中国科学院大学软件研究所);Alibaba Group(阿里巴巴集团);IBM Research(IBM研究院)
How Robust Is Multimodal Claim Verification to LLM Rewriting?
NII LLMC, Japan(国立情报学研究所 LLMC,日本);National Institute of Informatics, Japan(国立情报学研究所,日本);University of Göttingen, Germany(哥廷根大学,德国);University of Bologna, Italy(博洛尼亚大学,意大利);The University of Tokyo, Japan(东京大学,日本);Inria, LS2N, Nantes Université, France(法国国家信息与自动化研究所,LS2N,南特大学,法国)
Improving Constituency Parsing with Span Attention
University of Washington(华盛顿大学);The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院)
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Facebook AI Research(Facebook人工智能研究院);University of Washington(华盛顿大学);Princeton University(普林斯顿大学);Allen Institute of Artificial Intelligence(艾伦人工智能研究所)
SEPT: Improving Scientific Named Entity Recognition with Span Representation
我们引入了一个新的科学命名实体识别器,称为SEPT,它代表Span Extractor with Pre-trained Transformers。在最近的论文中,跨度提取器已被证明是一个强大的模型相比,序列标记模型。然而,我们发现,随着预训练的语言模型的发展,跨度提取器的性能似乎变得与序列标记模型相似。为了保持跨度表示的优点,我们对模型进行了欠采样,以平衡正负样本,减少搜索空间。此外,我们简化了原始网络结构,将跨度提取器与BERT结合起来。实验表明,即使是简化的体系结构也能达到相同的性能,SEPT在不涉及关系信息的情况下也能实现科学的命名实体识别,达到了一个新的技术水平。
AIxcellent Vibes at GermEval 2025 Shared Task on Candy Speech Detection: Improving Model Performance by Span-Level Training
Department of Medical Engineering and Technomathematics(医学工程与技术学系);Institute for Data-Driven Technologies(数据驱动技术研究所);FH Aachen University of Applied Sciences(亚琛应用科学大学);Academic and Research Department Engineering Hydrology(工程水文学学术与研究部门);IT Center RWTH Aachen University(亚琛工业大学IT中心);Department for Data Science and AI(数据科学与人工智能部门);ORDIX AG(ORDIX公司);Department of Information and Computing Sciences(信息与计算科学系)
Learning to Substitute Spans towards Improving Compositional Generalization
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);City University of Hong Kong(香港城市大学)
Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Kuaishou Technology(快手科技)
Span-level Detection of AI-generated Scientific Text via Contrastive Learning and Structural Calibration
Beijing Renhe Information Technology Co., Ltd.(北京润和信息技术有限公司);Key Laboratory of Digital Publishing and Total Process Management of Scientific and Technical Journals(科技期刊数字出版与全流程管理重点实验室)