Calibrating Artificial Guilt: Neurally Grounded Reward Shaping for Prosocial Multi-Agent Reinforcement Learning

2026/08/05 by Aaditya Mehta, Arya Shah
Computer Science · #cs.AI

paper · pdf

12 pages, 6 figures, 3 tables

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

Cooperative multi-agent reinforcement learning often adds social terms to individual rewards, yet the scale of those terms is usually chosen by hand. We ask whether a guilt signal can instead be calibrated from human neural and behavioural data and transferred to artificial agents. Using the public SoDec responsibility fMRI dataset (40 participants), we fit a subject-fixed-effects regression of momentary-happiness changes on outcome-type counts and recover a guilt weight as the Partner-negative minus Social-negative contrast (w=1.118, Cohen's d=0.214). We embed this weight in a two-agent Social Lottery environment and train independent Proximal Policy Optimization actor-critics under four shaping regimes: neurally calibrated, uniform constant, zero (selfish), and a unit-coefficient oracle. Across 1,000 evaluation episodes per condition, the calibrated agents track the human Social safe-choice rate most closely (0.459 vs. human 0.484; KL=0.0012), while the other three conditions deviate by one to three orders of magnitude in KL. Human neurobehavioural priors can therefore act as quantitative constraints on prosocial reward shaping.

Citations

• Gymnasium: A Standard Interface for Reinforcement Learning Environments
• Affective Coherence Monitoring for Transformer-Based Language Models
• Training language models to follow instructions with human feedback
• Human-centered mechanism design with Democratic AI
• The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
• Open Problems in Cooperative AI
• PettingZoo: Gym for Multi-Agent Reinforcement Learning
• Deep Reinforcement Learning and its Neuroscientific Implications
• Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Learning Reciprocity in Complex Sequential Social Dilemmas
• Scalable agent alignment via reward modeling: a research direction
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Exploration by Random Network Distillation
• Inequity aversion improves cooperation in intertemporal social dilemmas
• Diversity is All You Need: Learning Skills without a Reward Function
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
• Learning with Opponent-Learning Awareness
• Prosocial learning agents solve generalized Stag Hunts better than selfish ones
• Proximal Policy Optimization Algorithms
• A multi-agent reinforcement learning model of common-pool resource appropriation
• Maintaining cooperation in complex social dilemmas using deep reinforcement learning
• Deep reinforcement learning from human preferences
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Curiosity-driven Exploration by Self-supervised Prediction
• Multi-agent Reinforcement Learning in Sequential Social Dilemmas
• Towards an integration of deep learning and neuroscience
• Cooperative Inverse Reinforcement Learning
• Potential-Based Shaping and Q-Value Initialization are Equivalent