Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity

Miconi T
Rawal A
Clune J
Stanley KO
The impressive lifelong learning in animal brains is primarily enabled byplastic changes in synaptic connectivity. Importantly, these changes arenot passive, but are actively controlled by neuromodulation, which is itselfunder the control of the brain. The resulting self-modifying abilities of thebrain play an important role in learning and adaptation, and are a majorbasis for biological reinforcement learning. Here we show for the first timethat artificial neural networks with such neuromodulated plasticity can betrained with gradient descent. Extending previous work on differentiableHebbian plasticity, we propose a differentiable formulation for the neuro-modulation of plasticity. We show that neuromodulated plasticity improvesthe performance of neural networks on both reinforcement learning andsupervised learning tasks. In one task, neuromodulated plastic LSTMswith millions of parameters outperform standard LSTMs on a benchmarklanguage modeling task (controlling for the number of parameters). Weconclude that differentiable neuromodulation of plasticity offers a powerfulnew framework for training neural networks.
Pub. Info: 
International Conference on Learning Systems (ICLR) (31% acceptance rate)