Probabilistically Guaranteed Satisfaction of Temporal Logic Constraints During Reinforcement Learning

2021/02/19 by Derya Aksaray, Aksaray, Derya, Yasin Yazıcıoğlu +3 · 2 citations
Computer Science · #FOS: Computer and information sciences #FOS: Electrical engineering #Formal Methods in Verification #Optimization and Search Problems #Reinforcement Learning in Robotics #Robotics (cs.RO) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2102.10063

openalex publication_date 2021/02/19 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

We propose a novel constrained reinforcement learning method for finding optimal policies in Markov Decision Processes while satisfying temporal logic constraints with a desired probability throughout the learning process. An automata-theoretic approach is proposed to ensure the probabilistic satisfaction of the constraint in each episode, which is different from penalizing violations to achieve constraint satisfaction after a sufficiently large number of episodes. The proposed approach is based on computing a lower bound on the probability of constraint satisfaction and adjusting the exploration behavior as needed. We present theoretical results on the probabilistic constraint satisfaction achieved by the proposed approach. We also numerically demonstrate the proposed idea in a drone scenario, where the constraint is to perform periodically arriving pick-up and delivery tasks and the objective is to fly over high-reward zones to simultaneously perform aerial monitoring.

Cited by

• Motion Planning Under Temporal Logic Specifications In Semantically Unknown Environments
• Hyperproperty-Constrained Secure Reinforcement Learning

Related

• Reinforcement Learning Under Probabilistic Spatio-Temporal Constraints with Time Windows
• Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching
• Robust Satisfaction of Temporal Logic Specifications via Reinforcement Learning
• Reinforcement Learning for Temporal Logic Control Synthesis with\n Probabilistic Satisfaction Guarantees
• Reinforcement Learning with Temporal Logic Constraints for Partially-Observable Markov Decision Processes