2022/08/11 by Fernando de Meer Pardo, Pardo, Fernando de Meer, Christoph Auth +3
Business, Management and Accounting · Computer Science · Decision Sciences · #Auction Theory and Applications #Computational Engineering #FOS: Computer and information sciences #Finance #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Supply Chain and Inventory Management #and Science (cs.CE)
paper · pdf · doi:10.48550/arxiv.2208.06244
openalex publication_date 2022/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this article, we develop a modular framework for the application of Reinforcement Learning to the problem of Optimal Trade Execution. The framework is designed with flexibility in mind, in order to ease the implementation of different simulation setups. Rather than focusing on agents and optimization methods, we focus on the environment and break down the necessary requirements to simulate an Optimal Trade Execution under a Reinforcement Learning framework such as data pre-processing, construction of observations, action processing, child order execution, simulation of benchmarks, reward calculations etc. We give examples of each component, explore the difficulties their individual implementations & the interactions between them entail, and discuss the different phenomena that each component induces in the simulation, highlighting the divergences between the simulation and the behavior of a real market. We showcase our modular implementation through a setup that, following a Time-Weighted Average Price (TWAP) order submission schedule, allows the agent to exclusively place limit orders, simulates their execution via iterating over snapshots of the Limit Order Book (LOB), and calculates rewards as the $ improvement over the price achieved by a TWAP benchmark algorithm following the same schedule. We also develop evaluation procedures that incorporate iterative re-training and evaluation of a given agent over intervals of a training horizon, mimicking how an agent may behave when being continuously retrained as new market data becomes available and emulating the monitoring practices that algorithm providers are bound to perform under current regulatory frameworks.