This paper presents a multi-agent reinforcement learning algorithm to represent strategic bidding behavior in freight transport markets. Using this algorithm, we investigate whether feasible market equilibriums arise without any central control or communication between agents. Studying behavior in such environments may serve as a stepping stone towards self-organizing logistics systems like the Physical Internet. We model an agent-based environment in which a shipper and a carrier actively learn bidding strategies using policy gradient methods, posing bid- and ask prices at the individual container level. Both agents aim to learn the best response given the expected behavior of the opposing agent. A neutral broker allocates jobs based on bid-ask spreads.
Our game-theoretical analysis and numerical experiments focus on behavioral insights. To evaluate system performance, we measure adherence to Nash equilibria, fairness of reward division and utilization of transport capacity. We observe good performance both in predictable, deterministic settings (~95% adherence to Nash equilibria) and highly stochastic environments (~85% adherence). Risk-seeking behavior may increase an agent's reward share, as long as the strategies are not overly aggressive. The results suggest a potential for full automation and decentralization of freight transport markets.
Smart modular freight containers -- as propagated in the Physical Internet paradigm -- are equipped with sensors, data storage capability and intelligence that enable them to route themselves from origin to destination without manual intervention or central governance. In this self-organizing setting, containers can autonomously place bids on transport services in a spot market setting. However, for individual containers it may be difficult to learn good bidding policies due to limited observations. By sharing information and costs between one another, smart containers can jointly learn biddin…
Alibaba Group（阿里巴巴集团）；University College London（伦敦大学学院）；Shanghai JiaoTong University（上海交通大学）
Real-time advertising allows advertisers to bid for each impression for a visiting user. To optimize specific goals such as maximizing revenue and return on investment (ROI) led by ad placements, advertisers not only need to estimate the relevance between the ads and user's interests, but most importantly require a strategic response with respect to other advertisers bidding in the market. In this paper, we formulate bidding optimization with multi-agent reinforcement learning. To deal with a large number of advertisers, we propose a clustering method and assign each cluster with a strategic b…
Optimizing Search Advertising Strategies: Integrating Reinforcement Learning with Generalized Second-Price Auctions for Enhanced Ad Ranking and Bidding
Columbia University（哥伦比亚大学）；University of Michigan（密歇根大学）；University of California, Irvine（加利福尼亚大学尔湾分校）
Shanghai Jiao Tong University（上海交通大学）；University College London（伦敦大学学院）；MediaGamma Ltd（MediaGamma有限公司）；Vlion Inc（Vlion公司）
The majority of online display ads are served through real-time bidding (RTB) --- each ad display impression is auctioned off in real-time when it is just being generated from a user visit. To place an ad automatically and optimally, it is critical for advertisers to devise a learning algorithm to cleverly bid an ad impression in real-time. Most previous works consider the bid decision as a static optimization problem of either treating the value of each impression independently or setting a bid price to each segment of ad volume. However, the bidding for a given ad campaign would repeatedly h…
The large integration of variable energy resources is expected to shift a large part of the energy exchanges closer to real-time, where more accurate forecasts are available. In this context, the short-term electricity markets and in particular the intraday market are considered a suitable trading floor for these exchanges to occur. A key component for the successful renewable energy sources integration is the usage of energy storage. In this paper, we propose a novel modelling framework for the strategic participation of energy storage in the European continuous intraday market where exchange…
Convergence bidding has been adopted in recent years by most Independent System Operators (ISOs) in the United States as a relatively new market mechanism to enhance market efficiency. Convergence bidding affects many aspects of the operation of the electricity markets and there is currently a gap in the literature on understanding how the market participants strategically select their convergence bids in practice. To address this open problem, in this paper, we study three years of real-world market data from the California ISO energy market. First, we provide a data-driven overview of all su…
Learning Safe Strategies for Value Maximizing Buyers in Uniform Price Auctions
Sloan School of Management, Massachusetts Institute of Technology（麻省理工学院斯隆管理学院）；Operations Research Center, Massachusetts Institute of Technology（麻省理工学院运筹学研究中心）
Convergence bidding, a.k.a., virtual bidding, has been widely adopted in wholesale electricity markets in recent years. It provides opportunities for market participants to arbitrage on the difference between the day-ahead market locational marginal prices and the real-time market locational marginal prices. Given the fact that convergence bids (CBs) have a significant impact on the operation of electricity markets, it is important to understand how market participants strategically select their CBs in real-world. We address this open problem with focus on the electricity market that is operat…
