Neuronlike adaptive elements that can solve difficult learning control problems

1983/09/01 by Andrew G. Barto, Richard S. Sutton, Charles W. Anderson · 106 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · Mathematics · #Adaptive Dynamic Programming Control #Adaptive control #Artificial intelligence #Associative property #Computer science #Control (management) #Element (criminal law) #Engineering #Function (biology) #Mathematics #Q-learning #Reinforcement #Reinforcement learning #Relation (database) #Task (project management) #Zebrafish Biomedical Research Applications

paper · doi:10.1109/tsmc.1983.6313077

published in IEEE Transactions on Systems Man and Cybernetics SMC-13(5), 834-846 (Institute of Electrical and Electronics Engineers)

openalex publication_date 1983/09/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/23

Abstract

It is shown how a system consisting of two neuronlike adaptive elements can solve a difficult learning control problem. The task is to balance a pole that is hinged to a movable cart by applying forces to the cart's base. It is argued that the learning problems faced by adaptive elements that are components of adaptive networks are at least as difficult as this version of the pole-balancing problem. The learning system consists of a single associative search element (ASE) and a single adaptive critic element (ACE). In the course of learning to balance the pole, the ASE constructs associations between input and output by searching under the influence of reinforcement feedback, and the ACE constructs a more informative evaluation function than reinforcement feedback alone can provide. The differences between this approach and other attempts to solve problems using neurolike elements are discussed, as is the relation of this work to classical and instrumental conditioning in animal learning studies and its possible implications for research in the neurosciences.

Cited by

• On Optimal Event-Triggered Distributed Control for Stochastic Multi-Agent Systems via Reinforcement Learning
• The Open Ant: A Robot Platform for Reinforcement Learning Research
• Understanding Reinforcement Learning for Model Training, and future directions with GRAPE
• Replicable Reinforcement Learning with Linear Function Approximation
• Memoryless Policy Iteration for Episodic POMDPs
• Reinforcement Learning in POMDP's via Direct Gradient Ascent
• Outcome-Aware Spectral Feature Learning for Instrumental Variable Regression
• Expressive Temporal Specifications for Reward Monitoring
• Multi-Agent Craftax: Benchmarking Open-Ended Multi-Agent Reinforcement Learning at the Hyperscale
• Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction
• The Confusing Instance Principle for Online Linear Quadratic Control
• Thompson Sampling via Fine-Tuning of LLMs
• Reinforcement Learning with Action-Triggered Observations
• Control Synthesis of Cyber-Physical Systems for Real-Time Specifications through Causation-Guided Reinforcement Learning
• Sample-Efficient and Smooth Cross-Entropy Method Model Predictive Control Using Deterministic Samples
• TDHook: A Lightweight Framework for Interpretability
• Sampling Complexity of TD and PPO in RKHS
• Quantifying Generalisation in Imitation Learning
• Online reinforcement learning via sparse Gaussian mixture model Q-functions
• Neuromorphic Intelligence
• Deep Reinforcement Learning for Ranking Utility Tuning in the Ad Recommender System at Pinterest
• QTMRL: An Agent for Quantitative Trading Decision-Making Based on Multi-Indicator Guided Reinforcement Learning
• HAEPO: History-Aggregated Exploratory Policy Optimization
• Dynamical Alignment: A Principle for Adaptive Neural Computation
• Distilling Reinforcement Learning into Single-Batch Datasets
• Hardware-Aware Fine-Tuning of Spiking Q-Networks on the SpiNNaker2 Neuromorphic Platform
• Time-Aware World Model for Adaptive Prediction and Control
• Control-Optimized Deep Reinforcement Learning for Artificially Intelligent Autonomous Systems
• Reusing Trajectories in Policy Gradients Enables Fast Convergence
• Is there Value in Reinforcement Learning?
• A Differential Perspective on Distributional Reinforcement Learning
• Neural Path Guiding with Distribution Factorization
• Bellman operator convergence enhancements in reinforcement learning algorithms
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
• Bi-Level Policy Optimization with Nyström Hypergradients
• Evolution imposes an inductive bias that alters and accelerates learning dynamics
• Model Identification Adaptive Control with ρ-POMDP Planning
• Continuous Thought Machines
• Wasserstein Policy Optimization
• AlphaCNOT: Learning CNOT Minimization with Model-Based Planning
• Revisiting TD Target Aggregation under Uncertainty in Q-Learning
• Efficient Tree Generation for Globally Optimal Decisions under Probabilistic Outcomes
• Episodically adapted network-based controllers
• Which is the best intrinsic motivation signal for learning multiple skills? [europepmc]
• Curiosity driven reinforcement learning for motion planning on humanoids. [europepmc]
• Absence of "Warm-Up" during Active Avoidance Learning in a Rat Model of Anxiety Vulnerability: Insights from Computational Modeling. [europepmc]
• The algorithmic anatomy of model-based evaluation. [europepmc]
• Morphological elucidation of basal ganglia circuits contributing reward prediction. [europepmc]
• The power of associative learning and the ontogeny of optimal behaviour. [europepmc]
• Reward-based training of recurrent neural networks for cognitive and value-based tasks. [europepmc]
• Neural activity in cortico-basal ganglia circuits of juvenile songbirds encodes performance during goal-directed learning. [europepmc]
• Regression techniques employing feature selection to predict clinical outcomes in stroke. [europepmc]
• Cerebellar learning using perturbations. [europepmc]
• Reward prediction error does not explain movement selectivity in DMS-projecting dopamine neurons. [europepmc]
• Motor Control System for Adaptation of Healthy Individuals and Recovery of Poststroke Patients: A Case Study on Muscle Synergies. [europepmc]
• The Cerebro-Cerebellum as a Locus of Forward Model: A Review. [europepmc]
• Dysmetria and Errors in Predictions: The Role of Internal Forward Model. [europepmc]
• Adolescents exhibit reduced Pavlovian biases on instrumental learning. [europepmc]
• Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task. [europepmc]
• Reinforcement Learning With Human Advice: A Survey. [europepmc]
• CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility. [europepmc]
• Engineering Spiking Neurons Using Threshold Switching Devices for High-Efficient Neuromorphic Computing. [europepmc]
• Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T. [europepmc]
• Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning. [europepmc]
• Deep Deterministic Policy Gradient-Based Autonomous Driving for Mobile Robots in Sparse Reward Environments. [europepmc]
• Effects of Stimulus Timing on the Acquisition of an Olfactory Working Memory Task in Head-Fixed Mice. [europepmc]
• Rise of artificial general intelligence: risks and opportunities. [europepmc]
• Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts. [europepmc]

Related

• Learning as Reinforcement: Applying Principles of Neuroscience for More General Reinforcement Learning Agents
• The brain as an efficient and robust adaptive learner
• Training Reinforcement Neurocontrollers Using the Polytope Algorithm
• Learning with incomplete information - and the mathematical structure behind it
• Optimal Neuron Selection: NK Echo State Networks for Reinforcement Learning