A Reinforcement Learning Formulation of the Lyapunov Optimization: Application to Edge Computing Systems with Queue Stability

2020/12/14 by Sohee Bae, Bae, Sohee, Seungyul Han +3 · 1 citation
Computer Science · Engineering · #Age of Information Optimization #Smart Grid Energy Management #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2012.07279

Abstract

In this paper, a deep reinforcement learning (DRL)-based approach to the Lyapunov optimization is considered to minimize the time-average penalty while maintaining queue stability. A proper construction of state and action spaces is provided to form a proper Markov decision process (MDP) for the Lyapunov optimization. A condition for the reward function of reinforcement learning (RL) for queue stability is derived. Based on the analysis and practical RL with reward discounting, a class of reward functions is proposed for the DRL-based approach to the Lyapunov optimization. The proposed DRL-based approach to the Lyapunov optimization does not required complicated optimization at each time step and operates with general non-convex and discontinuous penalty functions. Hence, it provides an alternative to the conventional drift-plus-penalty (DPP) algorithm for the Lyapunov optimization. The proposed DRL-based approach is applied to resource allocation in edge computing systems with queue stability and numerical results demonstrate its successful operation.

Cited by

• Splitwise: Collaborative Edge-Cloud Inference for LLMs via Lyapunov-Assisted DRL

Related

• A Lyapunov Drift-Plus-Penalty Method Tailored for Reinforcement Learning with Queue Stability
• A Structure-aware Online Learning Algorithm for Markov Decision Processes
• Stable Reinforcement Learning with Unbounded State Space
• Off Policy Lyapunov Stability in Reinforcement Learning
• Convex Programs and Lyapunov Functions for Reinforcement Learning: A Unified Perspective on the Analysis of Value-Based Methods