Reinforcement learning is used to align language models with human preference signals after first pre-training the model to predict the next token of text within a large corpus using likelihood maximization. Before being deployed in a specific domain, models are often further fine-tuned on task specific data. Since human preferences are often unavailable for the last step, it is performed using likelihood maximization as that is the typical default method. However, reinforcement learning has other advantages besides facilitating alignment to a human derived reward function. For one, whereas likelihood maximization is a form of imitation learning in which the model is trained on what to do under ideal conditions, reinforcement learning is not limited to demonstrating actions just for optimally reached states and trains a model what to do under a range of scenarios as it explores the policy space. In addition, it also trains a model what not to do, suppressing competitive but poor actions. This work develops a framework for last-mile fine-tuning using reinforcement learning and tests whether it garners performance gains. The experiments center on abstractive summarization, but the framework is general and broadly applicable. Use of the procedure produced significantly better results than likelihood maximization when comparing raw predictions. For the specific data tested, the gap could be bridged by employing post-processing of the maximum likelihood outputs. Nonetheless, the framework offers a new avenue for model optimization in situations where post-processing may be less straightforward or effective, and it can be extended to include more complex classes of undesirable outputs to penalize and train against, such as hallucinations.
Meta AI Research(Meta人工智能研究院);Carnegie Mellon University(卡内基梅隆大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
像图像-目标导航这样的现实长期任务涉及探索和利用阶段。由于目标的图像,一个具体的代理必须探索发现目标,即,使用学习的先验知识进行有效搜索。一旦目标被发现,智能体必须准确地校准到目标的最后一英里的导航。与任何强大的系统一样,探索性目标发现和利用性最后一英里导航之间的切换可以更好地从错误中恢复。根据这些直观的导轨,我们提出了SLING,以提高现有的图像目标导航系统的性能。作为对现有方法的补充,我们专注于最后一英里导航,并利用神经描述符来利用问题的底层几何结构。通过简单但有效的开关,我们可以轻松地将SLING与启发式、强化学习和神经模块化策略联系起来。在标准化的图像目标导航基准(Hahn et al. 2021)上,我们提高了策略,场景和情节复杂性的性能,将最先进的成功率从45%提高到55%。除了真实感模拟,我们在三个物理场景中进行了真实的机器人实验,并发现这些改进可以很好地转移到真实环境中。
Integrated Order Dispatching and Routing for Last-Mile Pickup via Deep Reinforcement Learning
College of Management and Economics, Tianjin University(天津大学管理与经济学部);Laboratory of Computation and Analytics of Complex Management Systems (CACMS), Tianjin University(天津大学复杂管理系统计算与分析实验室)
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging
Shanghai Jiao Tong University(上海交通大学);University of California, Davis(加州大学戴维斯分校);East China Normal University(华东师范大学);University of Pennsylvania(宾夕法尼亚大学)
Efficient Uncoupled Learning Dynamics with $\tilde{O}\!\left(T^{-1/4}\right)$ Last-Iterate Convergence in Bilinear Saddle-Point Problems over Convex Sets under Bandit Feedback
University of Washington(华盛顿大学);University of California, Irvine(加州大学尔湾分校)
DeepLethe;Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Beihang University(北京航空航天大学);Xiamen University(厦门大学);Nanjing University(南京大学);Beijing Jiaotong University(北京交通大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tianjin University(天津大学);Shanghai University of Finance and Economics(上海财经大学);Harvard University(哈佛大学);Stanford University(斯坦福大学);University College London(伦敦大学学院);Cornell University(康奈尔大学);Huawei(华为);Atlassian;Meta;Microsoft Research Asia(微软亚洲研究院);UWits Technology (Beijing) Co., Ltd.(优维思科技(北京)有限公司);Bopai Management Consulting (Shenzhen) Co., Ltd.(博湃管理咨询(深圳)有限公司);Exploration and Development Research Institute, PetroChina Southwest Oil & Gasfield Company(中国石油西南油气田公司勘探开发研究院);Kunlun Digital Techn(昆仑数字技术)