Photo of A biologically plausible technique for training a neural net

A biologically plausible technique for training a neural net

by Kurt Kleiner News Learning in Machines & Brains 17.05.2017

Artificial intelligence researchers have made advances in recent years by designing neural networks that are based on the structure of the human brain. But their work isn’t just leading to better computers – it is also leading to better understanding about how the neural networks in our own brains work.

Learning in Machines & Brains Co-Director Yoshua Bengio and his student Benjamin Scellier, both of the Université de Montréal, invented a new way of training artificial neural networks that might help theoretical neuroscientists figure out how natural neural networks learn and correct errors.

They call the technique “equilibrium propagation,” and it is an alternative to a widely used technique for training neural networks called backpropagation.

To train an artificial neural network with backpropagation you first present it with an input, propagate the signal forward in the network, then examine the output and compare it to the output you would ideally like to have gotten. The difference between the two is called the error. In the second step you “backpropagate” the error through the network, making adjustments to individual neurons and synapses in an attempt to get the output closer to the ideal. By repeating the process many times you gradually reduce the error in the neural network, bringing it closer to the output you want.

These illustrations represent a network in which equilibrium propagation works, left, with many recurrent and symmetric connections among nodes. On the right is a layered network of the kind in which backpropagation is useful.

But the way backpropagation works in artificial neural networks has never seemed biologically plausible, says Scellier. One reason among many others is that it requires a special computational circuit for the backpropagation of errors, which seems unlikely to have arisen in an evolved organism, based on current knowledge in neuroscience.

The new equilibrium propagation technique is possible using a single circuit, and a single type of calculation, says Scellier. “Our model requires only one type of neuronal dynamics to perform both inference and backpropagation of errors. The computations executed in the network are based on a standard neuron model and a standard form of synaptic plasticity.” That means it’s more likely to be similar to an actual process that evolved in the brain, and could provide the beginnings of an answer to how biological neural circuits learn.

That’s one of the major goals of the work, Scellier says.

“Today, the gap between neuroscience and the neural networks used in artificial intelligence is pretty big. Our approach is to start from a model with good machine learning properties, and gradually add details that make the model more biologically realistic,” he says.

The next step will be for neuroscientists to design experiments to see if the brain itself uses similar techniques.

The paper “Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation” was published in Frontiers in Computational Neuroscience.

Related Ideas

News | Bio-inspired Solar Energy | CIFAR Azrieli Global Scholars

Synthetic circuit mimics photosynthesis light-harvesting efficiency

Researchers at Harvard, MIT and Arizona State University have designed a new synthetic structure that can harvest light energy in...

Video | Integrated Microbial Biodiversity

Marine Microbes – Our Invisible Allies

“We tend to think of things that we can see as being the really important contributors to the environment …...

Video | Integrated Microbial Biodiversity

Collaborative Field-Based Research: CIFAR’s program in Integrated Microbial Biodiversity

“You never know what you’re going to find… that’s the unpredictability aspect of all these expeditions,” says Senior Fellow Julius...

News | Integrated Microbial Biodiversity

Newly discovered phytoplankton groups appear to favour warmer oceans

An international research team has discovered two phytoplankton groups – unlike any known species – in climate-sensitive areas around the...

News | Integrated Microbial Biodiversity

First photos emerge of elusive marine predators

A global team of CIFAR researchers has captured the first images of a microbe that is one of the most abundant predators in the ocean, yet has until now remained unseen.

News | Integrated Microbial Biodiversity

Surprise third party found in lichen symbiosis

Since the discovery of their true nature 140 years ago, lichens have been the poster children for symbiosis. In the...