Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL

2020/12/17 by Simon Hirlaender, Hirlaender, Simon, Niky Bruchon +1 · 2 citations
Decision Sciences · Engineering · Physics and Astronomy · #Accelerator Physics (physics.acc-ph) #Advancements in Semiconductor Devices and Circuit Design #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Physical sciences #I.2 #J.2 #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Reservoir Engineering and Simulation Methods #Scientific Computing and Data Management #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2012.09737

openalex publication_date 2020/12/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning holds tremendous promise in accelerator controls. The primary goal of this paper is to show how this approach can be utilised on an operational level on accelerator physics problems. Despite the success of model-free reinforcement learning in several domains, sample-efficiency still is a bottle-neck, which might be encompassed by model-based methods. We compare well-suited purely model-based to model-free reinforcement learning applied to the intensity optimisation on the FERMI FEL system. We find that the model-based approach demonstrates higher representational power and sample-efficiency, while the asymptotic performance of the model-free method is slightly superior. The model-based algorithm is implemented in a DYNA-style using an uncertainty aware model, and the model-free algorithm is based on tailored deep Q-learning. In both cases, the algorithms were implemented in a way, which presents increased noise robustness as omnipresent in accelerator control problems. Code is released in https://github.com/MathPhysSim/FERMI_RL_Paper.

Cited by

• Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seeking
• 2022 Review of Data-Driven Plasma Science

Related

• Optimisation of the Accelerator Control by Reinforcement Learning: A Simulation-Based Approach
• Explainable physics-based constraints on reinforcement learning for accelerator controls
• Reinforcement Learning for Accelerator Beamline Control: a simulation-based approach
• Autonomous Control of a Particle Accelerator using Deep Reinforcement Learning
• Developing Robust Digital Twins and Reinforcement Learning for Accelerator Control Systems at the Fermilab Booster