2024/04/03 by Jules Hedges, Hedges, Jules, Riu Rodríguez Sakamoto +1 · 1 citation
Engineering · #Advanced Research in Systems and Signal Processing #Category Theory (math.CT) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2404.02688
openalex publication_date 2024/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We show that several major algorithms of reinforcement learning (RL) fit into the framework of categorical cybernetics, that is to say, parametrised bidirectional processes. We build on our previous work in which we show that value iteration can be represented by precomposition with a certain optic. The outline of the main construction in this paper is: (1) We extend the Bellman operators to parametrised optics that apply to action-value functions and depend on a sample. (2) We apply a representable contravariant functor, obtaining a parametrised function that applies the Bellman iteration. (3) This parametrised function becomes the backward pass of another parametrised optic that represents the model, which interacts with an environment via an agent. Thus, parametrised optics appear in two different ways in our construction, with one becoming part of the other. As we show, many of the major classes of algorithms in RL can be seen as different extremal cases of this general setup: dynamic programming, Monte Carlo methods, temporal difference learning, and deep RL. We see this as strong evidence that this approach is a natural one and believe that it will be a fruitful way to think about RL in the future.