Controlled automated story generation seeks to generate natural language stories satisfying constraints from natural language critiques or preferences. Existing methods to control for story preference utilize prompt engineering which is labor intensive and often inconsistent. They may also use logit-manipulation methods which require annotated datasets to exist for the desired attributes. To address these issues, we first train a contrastive bi-encoder model to align stories with corresponding human critiques, named CARP, building a general purpose preference model. This is subsequently used as a reward function to fine-tune a generative language model via reinforcement learning. However, simply fine-tuning a generative language model with a contrastive reward model does not always reliably result in a story generation system capable of generating stories that meet user preferences. To increase story generation robustness we further fine-tune the contrastive reward model using a prompt-learning technique. A human participant study is then conducted comparing generations from our full system, ablations, and two baselines. We show that the full fine-tuning pipeline results in a story generator preferred over a LLM 20x as large as well as logit-based methods. This motivates the use of contrastive learning for general purpose human preference modeling.
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science and Electronic Engineering, University of Surrey（Surrey大学计算机科学与电子工程学院）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Nanjing University（南京大学）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
Artefact Research Center（Artefact 研究中心）；Unbabel；Equall；MICS, CentraleSupélec, Université Paris-Saclay（巴黎-萨克雷大学中央理工-高等电力学院数学与计算机科学实验室）；Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico & Universidade de Lisboa（里斯本大学高等技术学院）
Random Direct Preference Optimization for Radiography Report Generation
Artificial Intelligence Research Institute (AIRI)（人工智能研究院）；Skolkovo Institute of Science and Technology（斯克罗夫学院）；Institute for Information Transmission Problems（信息传输问题研究所）
Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；LLM Team, Shopee Pte. Ltd.（Shopee Pte. Ltd. 语言模型团队）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance and Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（北京大型模型与智能治理研究重点实验室）
Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Center for Data Science, AAIS, Peking University（北京大学AAIS数据科学中心）；Ant Group（蚂蚁集团）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）
CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
Department of Computer Science (DIKU), University of Copenhagen（哥本哈根大学计算机科学系（DIKU））；Centre for Language Technology (CST), University of Copenhagen（哥本哈根大学语言技术中心）；Pioneer Centre for Artificial Intelligence（先锋人工智能中心）
StoryER: Automatic Story Evaluation via Ranking, Rating and Reasoning
The University of Tokyo（东京大学）；Tokyo Institute of Technology（东京工业大学）；National Institute of Advanced Industrial Science Technology（国立先进工业科学技术研究院）
现有的自动故事评价方法重视故事词汇层次的连贯性，偏离了人类的偏好。我们超越了这个限制，考虑一种新的\textbf{Story} \textbf{E}估价方法，模仿人类的偏好时，判断一个故事，即\textbf{StoryER}，其中包括三个子任务：\textbf{R} anking，\textbf{R} ating和\textbf{R} easoning。给定机器生成或人类编写的故事，StoryER要求机器输出1）与人类偏好对应的偏好评分，2）特定评级及其相应的置信度，以及3）各个方面的评论（例如，开放，性格塑造）。为了支持这些任务，我们引入了一个注释良好的数据集，包括（i）100 k排名的故事对;以及（ii）一组46 k评级和对故事各个方面的评论。我们在收集的数据集上微调Longformer-Encoder-Decoder（LED），编码器负责偏好评分和方面预测，解码器负责评论生成。我们的综合实验为每个任务提供了一个有竞争力的基准，显示出与人类偏好的高度相关性。此外，我们还见证了偏好评分、方面评分和评论的联合学习在每个任务中都带来了收益。我们的数据集和基准是公开的，以推进故事评估任务的研究。\footnote{Dataset and pre-trained model demo are available at anonymous website \url{http://story…
Personalized Text Generation with Contrastive Activation Steering
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Northeastern University（东北大学）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）
HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models
Zhejiang University（浙江大学）；Byte Dance（字节跳动）；National University of Singapore（新加坡国立大学）；Angelalign Inc China（Angelalign中国公司）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）
