In this paper, we investigate the online parcel assignment (OPA) problem, in which each stochastically generated parcel needs to be assigned to a candidate route for delivery to minimize the total cost subject to certain business constraints. The OPA problem is challenging due to its stochastic nature: each parcel's candidate routes, which depends on the parcel's origin, destination, weight, etc., are unknown until its order is placed, and the total parcel volume is uncertain in advance. To tackle this challenge, we propose the PPO-OPA algorithm based on deep reinforcement learning that shows competitive performance. More specifically, we introduce a novel Markov Decision Process (MDP) framework to model the OPA problem, and develop a policy gradient algorithm that adopts attention networks for policy evaluation. By designing a dedicated reward function, our proposed algorithm can achieve a lower total cost with smaller violation of constraints, comparing to the traditional method which assigns parcels to candidate routes proportionally. In addition, the performances of our proposed algorithm and the Primal-Dual algorithm are comparable, while the later assumes a known total parcel volume in advance, which is unrealistic in practice.
Effective Online 3D Bin Packing with Lookahead Parcels Using Monte Carlo Tree Search
Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education China（高可信软件技术重点实验室（北京大学））；Faculty of Computing, Harbin Institute of Technology, Harbin, China（计算机学院，哈尔滨工业大学，哈尔滨，中国）；JD Logistic, Beijing, China（京东物流，北京，中国）；School of Computer Science, Peking University, Beijing, China（计算机科学学院，北京大学，北京，中国）
vivo AI Lab（vivo人工智能实验室）；Zhejiang Lab（浙江实验室）；CUHK MMLab（香港大学多模态实验室）；Hubei University（湖北省大学）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）
Minimal Ingredients for Reward Assignment from Expert Demonstrations
New York University（纽约大学）；New York University Abu Dhabi（纽约大学阿布扎赫尔分校）；Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning, NYU Shanghai（上海前沿科学中心（人工智能与深度学习））
Efficient credit assignment is essential for reinforcement learning algorithms in both prediction and control settings. We describe a unified view on temporal-difference algorithms for selective credit assignment. These selective algorithms apply weightings to quantify the contribution of learning updates. We present insights into applying weightings to value-based learning and planning algorithms, and describe their role in mediating the backward credit distribution in prediction and control. Within this space, we identify some existing online learning algorithms that can assign credit select…
University of Southern California（南加州大学）；General Motors Global R&D（通用汽车全球研发中心）
There has been a growing interest for Wireless Distributed Computing (WDC), which leverages collaborative computing over multiple wireless devices. WDC enables complex applications that a single device cannot support individually. However, the problem of assigning tasks over multiple devices becomes challenging in the dynamic environments encountered in real-world settings, considering that the resource availability and channel conditions change over time in unpredictable ways due to mobility and other factors. In this paper, we formulate a task assignment problem as an online learning problem…
The temporal Credit Assignment Problem (CAP) is a well-known and challenging task in AI. While Reinforcement Learning (RL), especially Deep RL, works well when immediate rewards are available, it can fail when only delayed rewards are available or when the reward function is noisy. In this work, we propose delegating the CAP to a Neural Network-based algorithm named InferNet that explicitly learns to infer the immediate rewards from the delayed rewards. The effectiveness of InferNet was evaluated on two online RL tasks: a simple GridWorld and 40 Atari games; and two offline RL tasks: GridWorld…
The challenge in the widely applicable online matching problem lies in making irrevocable assignments while there is uncertainty about future inputs. Most theoretically-grounded policies are myopic or greedy in nature. In real-world applications where the matching process is repeated on a regular basis, the underlying data distribution can be leveraged for better decision-making. We present an end-to-end Reinforcement Learning framework for deriving better matching policies based on trial-and-error on historical data. We devise a set of neural network architectures, design feature representati…
Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning
The Swiss AI Lab IDSIA, USI & SUPSI（瑞士IDSIA人工智能实验室、瑞士意大利语区大学、瑞士南部应用科学大学）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）；AI Initiative, King Abdullah University of Science and Technology（阿卜杜拉国王科技大学人工智能计划）
Model-Based Reinforcement Learning for Heterogeneous Multi-Robot Task Assignment Under Distribution Shifts
John A. Paulson School Of Engineering And Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Harvard Medical School（哈佛医学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Stanford University School of Medicine（斯坦福大学医学院）
MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment
King’s College London（伦敦国王学院）；Eindhoven University of Technology（埃因霍温理工大学）；University of Liverpool（利物浦大学）；University of California San Diego（加州大学圣地亚哥分校）
The online assignment problem plays an important role in operational research and computer science which is why immense attention has been given to improving its solution quality. Due to the incomplete information about the input, it is difficult for online algorithms to produce the optimal solution. The quality of the solution of an online algorithm is measured using a competitive ratio. No online deterministic algorithm can achieve a competitive ratio better than (2n-1). It has been shown that advice in online computation improves the lower bound of the competitive ratio of online problems. …
Online controlled experiments (A/B tests) have become the gold standard for learning the impact of new product features in technology companies. Randomization enables the inference of causality from an A/B test. The randomized assignment maps end users to experiment buckets and balances user characteristics between the groups. Therefore, experiments can attribute any outcome differences between the experiment groups to the product feature under experiment. Technology companies run A/B tests at scale -- hundreds if not thousands of A/B tests concurrently, each with millions of users. The large …
Carnegie Mellon University（卡内基梅隆大学）；California Institute of Technology（加州理工学院）；Google, Inc.（谷歌公司）
Which ads should we display in sponsored search in order to maximize our revenue? How should we dynamically rank information sources to maximize value of information? These applications exhibit strong diminishing returns: Selection of redundant ads and information sources decreases their marginal utility. We show that these and other problems can be formalized as repeatedly selecting an assignment of items to positions to maximize a sequence of monotone submodular functions that arrive one by one. We present an efficient algorithm for this general problem and analyze it in the no-regret model.…
California Institute of Technology（加州理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Google, Inc.（谷歌公司）；ETH Zurich（苏黎世联邦理工学院）；Duolingo（多邻国）
Which ads should we display in sponsored search in order to maximize our revenue? How should we dynamically rank information sources to maximize the value of the ranking? These applications exhibit strong diminishing returns: Redundancy decreases the marginal utility of each ad or information source. We show that these and other problems can be formalized as repeatedly selecting an assignment of items to positions to maximize a sequence of monotone submodular functions that arrive one by one. We present an efficient algorithm for this general problem and analyze it in the no-regret model. Our …
MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments
Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague（布拉格捷克技术大学捷克信息学、机器人与控制论研究所）；Czech Technical University in Prague（布拉格捷克技术大学）；University of Crete（克里特大学）；IACM-Forth（希腊研究与技术基金会伊阿基莫斯研究所）；Archimedes/Athena RC（阿基米德/雅典娜研究中心）
Structural credit assignment for recurrent learning is challenging. An algorithm called RTRL can compute gradients for recurrent networks online but is computationally intractable for large networks. Alternatives, such as BPTT, are not online. In this work, we propose a credit-assignment algorithm -- \algoname{} -- that approximates the gradients for recurrent learning in real-time using $O(n)$ operations and memory per-step. Our method builds on the idea that for modular recurrent networks, composed of columns with scalar states, it is sufficient for a parameter to only track its influence on…
Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks
Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Peng Cheng Laboratory（鹏城实验室）
TU Darmstadt（达姆施塔特工业大学）；Bilkent University（比尔肯特大学）；University of California Los Angeles（加利福尼亚大学洛杉矶分校）；University of Oxford（牛津大学）
In mobile crowdsourcing (MCS), mobile users accomplish outsourced human intelligence tasks. MCS requires an appropriate task assignment strategy, since different workers may have different performance in terms of acceptance rate and quality. Task assignment is challenging, since a worker's performance (i) may fluctuate, depending on both the worker's current personal context and the task context, (ii) is not known a priori, but has to be learned over time. Moreover, learning context-specific worker performance requires access to context information, which may not be available at a central enti…
We propose a new model for augmenting algorithms with predictions by requiring that they are formally learnable and instance robust. Learnability ensures that predictions can be efficiently constructed from a reasonable amount of past data. Instance robustness ensures that the prediction is robust to modest changes in the problem input, where the measure of the change may be problem specific. Instance robustness insists on a smooth degradation in performance as a function of the change. Ideally, the performance is never worse than worst-case bounds. This also allows predictions to be objective…
