2025/03/15 by Solvi Arnold, Reiji Suzuki, Takaya Arita
Computer Science · Social Sciences · #Evolutionary Algorithms and Applications #Language and cultural evolution #Reinforcement Learning in Robotics
paper · doi:10.1162/artl.a.464
openalex publication_date 2025/03/15 · openalex created_date 2026/07/29 · openalex updated_date 2026/07/31
Efficient learning is fundamental to intelligence. Popular AI learning methods still fall short of advanced biological learners in terms of efficiency, and hence, in intelligence. Given the inherent trade-off between efficiency and generality, we can understand this low efficiency as the Cost of Generality incurred by general-purpose AI learning algorithms. Focusing on Reinforcement Learning (RL), we identify the Reward Bottleneck as a key manifestation of this cost. Advanced biological intelligence learns efficiently from an information-rich stream of stimulus information, even when explicit feedback on behaviour quality is sparse or absent. Backpropagation-based AI learning methods, in contrast, rely on explicit scalar quality measures (reward in the case of RL) as the sole driver of the learning process. The low information bandwidth of the driving signal limits efficiency. This pattern echoes behaviourist theories of learning that have long been recognised as insufficient for explaining biological learning. We argue that nature circumvents this bottleneck by adapting learning abilities to specific task domains, thereby making it possible to effectively exploit task-relevant information streams other than reward. Such domain-adaptation includes (but is not limited to) the integration of assumptions about the world (priors) into the machinery of learning. We refer to the resulting bottleneck-free learning as High-Bandwidth Learning (HBL). However, evolving such forms of learning artificially has proven to be challenging, suggesting a gap in our understanding of their evolvability. We explain that from-scratch evolution of HBL is a needle-in-a-haystack problem, as it requires unlikely confluences of information-eliciting behaviours and neural circuitry capable of learning from that information. We then propose that reward-driven learning (as modelled by RL), while insufficient as a model of learning by itself, can be a powerful catalyst for evolution of HBL. Given prior existence of reward-driven learning in a population, mutations that allow non-reward information to bias the learning process in a beneficial manner ("leaks" in the reward bottleneck) accumulate, enabling gradual construction of neural pathways for deriving behaviour updates directly from non-reward information, without the reliance on unlikely confluences that obstructs from-scratch evolution of HBL. This scenario provides a biologically plausible pathway towards HBL. To explore this scenario computationally, we set up a population of neural networks with reward-driven learning modelled as Reinforcement Learning (A2C), and allow evolution to improve learning efficiency by integrating non-reward information into the learning process using a neuromodulatory connection weight update mechanism. On a minimalistic navigation task domain in continuous 2D space, evolved agents show a 300-fold increase in learning speed compared to pure RL agents. Dependence on reward is eliminated, resulting in agents that learn from non-reward information exclusively, using local neuromodulation-based connection weight updates only. We then explore scalability to a high-dimensional task domain. In physically simulated quadruped locomotion learning with individual bodies randomised along 16 dimensions of variation, evolved HBL agents outperform evolved and non-evolved RL baselines in both learning speed and performance, with local, reward-agnostic neuromodulation-based connection weight updates accounting for most of their learning progress. We discuss our findings in context of the Baldwin effect, autonomy, and the apparent generality of human intelligence. Code available from the project page at github.com/sovvel/hbl.