Artificial neural networks – loosely inspired by biological neural networks – are a series of connected layers of ‘artificial neurons’ (which receive, process, and transmit signals) in which the output of each layer serves as the input for the subsequent layer. The crucial task of generating the input for the next layer is done by applying a non-linear activation function. This requires a lot of computing power and circuitry, due to the transfer of data back and forth between two units (for storing data and external processing).

Now, the San Diego researchers have developed a nanoscale device which can efficiently carry out the activation function.

“Neural network computations in hardware get increasingly inefficient as the neural network models get larger and more complex,” said Professor Duygu Kuzum, an electrical engineer at the university’s school of engineering. “We developed a single nanoscale artificial neuron device that implements these computations in hardware in a very area- and energy-efficient way.”

Her device implements one of the most common activation functions in neural network training: a rectified linear unit. This is notable because it requires hardware that can undergo a gradual shift in resistance; this is exactly what Kuzum and her PhD student built into their device. Their artificial neuron can gradually switch from an insulating to conducting state through heat transfer; typically, this transition (Mott transition) is abrupt.

This shift in resistance takes place in an ultra-thin layer of vanadium dioxide. Above this layer is a nanowire heater made of titanium and gold. When current flows through the nanowire – rather than flowing through the material itself – the ultra-thin layer slowly heats, causing a controlled switch from an insulating to conducting state.

“This device architecture is very interesting and innovative,” said PhD candidate Sangheon Oh, first author of the Nature Nanotechnology paper. “In this case, we flow current through a nanowire on top of the material to heat it and induce a very gradual resistance change.”

Oh and Kuzum fabricated an array of these tiny devices and a synaptic device array, integrated the two arrays on a custom-printed circuit board, then connected them together.

They put their hardware to the test by processing an image using edge detection: an approach to machine learning which involves identifying the edges of objects in an image. This demonstrated that the hardware can perform operations essential for many types of deep neural network. It runs these essential computations using 100 to 1,000 times less energy and area than existing hardware.

They hope to scale up this technology to perform more complex tasks, such as facial recognition or object detection for autonomous vehicles.

“Right now, this is a proof of concept,” said Kuzum. “It’s a tiny system in which we only stacked one synapse layer with one activation layer. By stacking more of these together, you could make a more complex system for different applications.”