Discriminative Probing and Tuning for Text-to-Image Generation
National University of Singapore（新加坡国立大学）；Hong Kong Polytechnic University（香港理工大学）；Nanyang Technological University（南洋理工大学）；Skywork AI（天工AI）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Despite advancements in text-to-image generation (T2I), prior methods often face text-image misalignment problems such as relation confusion in generated images. Existing solutions involve cross-attention manipulation for better compositional understanding or integrating large language models for improved layout planning. However, the inherent alignment capabilities of T2I models are still inadequate. By reviewing the link between generative and discriminative modeling, we posit that T2I models' discriminative abilities may reflect their text-image alignment proficiency during generation. In this light, we advocate bolstering the discriminative abilities of T2I models to achieve more precise text-to-image alignment for generation. We present a discriminative adapter built on T2I models to probe their discriminative abilities on two representative tasks and leverage discriminative fine-tuning to improve their text-image alignment. As a bonus of the discriminative adapter, a self-correction mechanism can leverage discriminative gradients to better align generated images to text prompts during inference. Comprehensive evaluations across three benchmark datasets, including both in-distribution and out-of-distribution scenarios, demonstrate our method's superior generation performance. Meanwhile, it achieves state-of-the-art discriminative performance on the two discriminative tasks compared to other generative models.
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；National University of Singapore（新加坡国立大学）；Tencent（腾讯）
Recent works have shown promising results of prompt tuning in stimulating pre-trained language models (PLMs) for natural language processing (NLP) tasks. However, to the best of our knowledge, existing works focus on prompt-tuning generative PLMs that are pre-trained to generate target tokens, such as BERT. It is still unknown whether and how discriminative PLMs, e.g., ELECTRA, can be effectively prompt-tuned. In this work, we present DPT, the first prompt tuning framework for discriminative PLMs, which reformulates NLP tasks into a discriminative language modeling problem. Comprehensive exper…
Ping An Life Insurance Company of China, Ltd.（中国平安人寿保险股份有限公司）
Patronizing and condescending language (PCL) has a large harmful impact and is difficult to detect, both for human judges and existing NLP systems. At SemEval-2022 Task 4, we propose a novel Transformer-based model and its ensembles to accurately understand such language context for PCL detection. To facilitate comprehension of the subtle and subjective nature of PCL, two fine-tuning strategies are applied to capture discriminative features from diverse linguistic behaviour and categorical distribution. The system achieves remarkable results on the official ranking, including 1st in Subtask 1 …
Recent works show that pre-trained language models (PTLMs), such as BERT, possess certain commonsense and factual knowledge. They suggest that it is promising to use PTLMs as "neural knowledge bases" via predicting masked words. Surprisingly, we find that this may not work for numerical commonsense knowledge (e.g., a bird usually has two legs). In this paper, we investigate whether and to what extent we can induce numerical commonsense knowledge from PTLMs as well as the robustness of this process. To study this, we introduce a novel probing task with a diagnostic dataset, NumerSense, containi…
TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；College of Software, Jilin University（吉林大学软件学院）；Fudan University（复旦大学）；School of Mathematical and Computer Sciences, Heriot-Watt University（赫瑞-瓦特大学数学与计算机科学学院）；Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）
Beyond Completion: Probing Cumulative State Tracking to Predict LLM Agent Performance
Tohoku University（东北大学）；Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学）；Texas A&M University（德克萨斯农工大学）；Worcester Polytechnic Institute（伍斯特理工学院）
Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Zhejiang University（浙江大学）；University of Central Florida（中佛罗里达大学）；Illinois Institute of Technology（伊利诺伊理工大学）
EmCom-Diffusion: Probing Visual Reflection in Emergent Languages via Image Generation
Graduate School of Informatics, Kyoto University, Kyoto, Japan（京都大学信息学研究科）；Research Organization of Science and Technology, Ritsumeikan University, Shiga, Japan（立命馆大学科学技术研究机构）
New York University（纽约大学）；Xi’an Jiaotong University（西安交通大学）；WeChat AI（微信人工智能）；Virginia Tech（弗吉尼亚理工大学）
Script knowledge is critical for humans to understand the broad daily tasks and routine activities in the world. Recently researchers have explored the large-scale pre-trained language models (PLMs) to perform various script related tasks, such as story generation, temporal ordering of event, future event prediction and so on. However, it's still not well studied in terms of how well the PLMs capture the script knowledge. To answer this question, we design three probing tasks: inclusive sub-event selection, starting sub-event selection and temporal ordering to investigate the capabilities of P…
RSRA: Training-Free Probing of Representation Sensitivity for Efficient LoRA Rank Allocation
Dalian University of Technology（大连理工大学）；Northeastern University（东北大学）；Hebei Key Laboratory of Marine Perception Network and Data Processing（河北省海洋感知网络与数据处理重点实验室）；Nanjing Tech University（南京工业大学）；Institute of Intelligent Manufacturing（智能制造研究院）
Progress in pre-trained language models has led to a surge of impressive results on downstream tasks for natural language understanding. Recent work on probing pre-trained language models uncovered a wide range of linguistic properties encoded in their contextualized representations. However, it is unclear whether they encode semantic knowledge that is crucial to symbolic inference methods. We propose a methodology for probing linguistic information for logical inference in pre-trained language model representations. Our probing datasets cover a list of linguistic phenomena required by major s…
University of Lille（里尔大学）；INRIA Lille（法国国家信息与自动化研究所里尔分部）；CNRS（法国国家科学研究中心）；Centrale Lille（里尔中央理工学院）；IBM Research Europe（IBM欧洲研究院）
We probe pre-trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in-comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of-Cloze test). Our formulation produces optimistic results without any fine-tuning, which indic…
IT University of Copenhagen（哥本哈根信息技术大学）；LMU Munich（慕尼黑大学）
Making an informed choice of pre-trained language model (LM) is critical for performance, yet environmentally costly, and as such widely underexplored. The field of Computer Vision has begun to tackle encoder ranking, with promising forays into Natural Language Processing, however they lack coverage of linguistic tasks such as structured prediction. We propose probing to rank LMs, specifically for parsing dependencies in a given language, by measuring the degree to which labeled trees are recoverable from an LM's contextualized embeddings. Across 46 typologically and architecturally diverse LM…
D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models
The Ohio State University（俄亥俄州立大学）；University of Chinese Academy of Sciences（中国科学院大学）；Tongji University（同济大学）；Nanjing University（南京大学）；The University of Hong Kong（香港大学）；Peking University（北京大学）；Alibaba International Digital Commerce（阿里巴巴国际数字商务）
EiCAP: Beyond Fluency, Probing and Improving Emotional Intelligence in LLMs via Psychologically Grounded Multi-Turn Dialogue
Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University（卡塔尔计算研究所（QCRI），哈马德·本·卡伊夫大学）；University of Illinois Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）
When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last linear layer -- the "head"). It is well known that fine-tuning leads to better accuracy in-distribution (ID). However, in this paper, we find that fine-tuning can achieve worse accuracy than linear probing out-of-distribution (OOD) when the pretrained features are good and the distribution shift is large. On 10 distribution shift datasets (Breeds-Living17, Breeds-Entity30, DomainNet, CIFAR $\to$ STL, CIFAR10.1, FMoW, …
Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning
Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University（安徽省多模态认知计算重点实验室，计算机科学与技术学院，安徽大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Nanjing University of Science and Technology（南京理工大学）
National University of Singapore（新加坡国立大学）；Chinese Academy of Sciences（中国科学院）；SEA AI Lab（SEA AI实验室）
Contrastive self-supervised learning (CSL) has attracted increasing attention for model pre-training via unlabeled data. The resulted CSL models provide instance-discriminative visual features that are uniformly scattered in the feature space. During deployment, the common practice is to directly fine-tune CSL models with cross-entropy, which however may not be the best strategy in practice. Although cross-entropy tends to separate inter-class features, the resulting models still have limited capability for reducing intra-class feature scattering that exists in CSL models. In this paper, we in…
CyberAgent, Inc.（赛博艾坚特公司）；RIKEN（日本理化学研究所）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）
Discriminativeness is a desirable feature of image captions: captions should describe the characteristic details of input images. However, recent high-performing captioning models, which are trained with reinforcement learning (RL), tend to generate overly generic captions despite their high performance in various other criteria. First, we investigate the cause of the unexpectedly low discriminativeness and show that RL has a deeply rooted side effect of limiting the output words to high-frequency words. The limited vocabulary is a severe bottleneck for discriminativeness as it is difficult fo…
SynDoc: A Hybrid Discriminative-Generative Framework for Enhancing Synthetic Domain-Adaptive Document Key Information Extraction
The University of Western Australia（西澳大学）；The University of Melbourne（墨尔本大学）；The University of Sydney（悉尼大学）；Weill Cornell Medicine, Cornell University（韦尔·科恩医学中心，康奈尔大学）
International School of Information Science & Engineering, Dalian University of Technology（大连理工大学国际信息科学与工程学院）；Huawei Cloud & AI（华为云与人工智能）；College of Computer and Engineering, Shandong University of Science and Technology（山东科技大学计算机与工程学院）；SenseTime Computer Vision Research Group, The University of Sydney（悉尼大学商汤计算机视觉研究组）
Fine-grained object retrieval aims to learn discriminative representation to retrieve visually similar objects. However, existing top-performing works usually impose pairwise similarities on the semantic embedding spaces or design a localization sub-network to continually fine-tune the entire model in limited data scenarios, thus resulting in convergence to suboptimal solutions. In this paper, we develop Fine-grained Retrieval Prompt Tuning (FRPT), which steers a frozen pre-trained model to perform the fine-grained retrieval task from the perspectives of sample prompting and feature adaptation…
Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency
Czech Technical University in Prague（捷克布拉格技术大学）；National Technical University of Athens（雅典国立技术大学）；University of Crete（克里特大学）；Archimedes, Athena RC（雅典娜研究中心阿基米德研究所）；IACM-FORTH（希腊基础研究与技术基金会计算机科学研究所）；IARAI（国际人工智能研究所）；IIT, NCSR “Demokritos”（德谟克利特国家科学研究中心信息与电信研究所）
