Real-Time Biologically Realistic Neuromorphic Attention - Software

Members: Mikhail Burtsev, Borys Wrobel, Christoph Posch, Daniel Sonnleithner, Daniel B. Fasnacht, Francesco Galluppi, Giacomo Indiveri, Kevin Gurney, Ernst Niebur, Alexander Rast, Rodolphe Héliot, Sergio Davies, Sim Bamford, Sushmita Allam, Thomas Pfeil, Thomas Sharp, Tao Zhou, Zeno Jonke

The real-time biologically realistic neuromorphic attention group will attempt to generate working models of attention. We plan on focussing on integrated robotic models of attention with at least some biological realism. The proposed model uses a Bayesian predictor spiking neural network as its "core". We hope to try various different dynamic neural models and determine optimum dynamics and parameters for the control task.

Modelling will focus on the SpiNNaker chip as a network "substrate", with an active intent to use pre-processed data from the Selective Attention (SAC) chip to minimise the need for retinal subsampling of inputs.

Implementing neural networks using the SpiNNaker system uses the PyNN package which we will provide on USB sticks

No previous experience is required, however, a background in dynamic neural models, Bayesian networks, or neural attention and selection mechanisms in biology would be particularly useful.

Topics of Interest

There are basically 3 main topics of interest we (the SpiNNaker group) have considered; others please feel free to add more!

- Is a Bayesian model of attention an effective as well as biologically realistic one, and if so, what is the simplest spiking neural

network we can build to generate Bayesian-like behaviour?

- Can we create spiking network based attentional models to control behaviour in a robot engaged in real-time, real-world activity?

- What level of detail is necessary or desirable in the dynamic model of the neuron to capture the necessary attentional behaviour?

Projects

Please, suggest more projects here (by editing this page)

A simplified model of the basal ganglia

The basal ganglia appear to play a central role in coordinating and selecting actions. We will attempt to model this subsystem using a very simplified base unit (minicolumn-like) to represent the effects of an individual bayesian predictor. The model will use Izhikevich neurons to model most of the underlying dynamics (including burst neurons)

Controlled pick-and place

We will attempt to use an integrated retinal sensor/robotic arm to control pick-and-place of various randomly assorted components. The key task will be to recognise the object of interest, pick it out from the relevant array of objects, and place it in another location with specified orientation. This deceptively simple-sounding task should actually be quite challenging and requires a fair amount of attention-guided control.

Track or Saccade?

Object tracking and following is not particularly difficult for neural systems, however, smooth tracking of objects can be hard to achieve in practice, particularly if the system must also make occasional saccades away from a current target of interest. We will use a simple 2-object model to refine the Bayesian predictor to provide smooth tracking of existing objects but fast saccades to unexpected objects in the scene, possibly extending the model to a multi-object environment. One of the important tasks here which we expect to be some challenge is the establishment of good priors for the datasets (i.e. the environment). We also expect some experimentation to find the best network for Bayesian-like prediction behaviour.

The Experiment

After further discussion by the group, we decided upon the following specific experiment:

Real-Time Foveation

This experiment essentially combines features of the "Simplified Model of the Basal Ganglia" and "Track or Saccade" project proposals. The experiment has 3 phases:

Phase 1: Discriminate between, and correctly foveate, 2 static stimuli of equal salience, based upon the orientation of lines within the stimuli.

Phase 2: Upgrade the model to handle different values of salience including the case where the preferred object of attention is less salient than distractors.

Phase 3: Permit smooth predictive tracking with moving objects in the presence of distractors

We have developed a basic system architecture (Fig. 1) for the model.

The experimental setup is as follows (Fig. 2)

We use a screen-based image, generated in whatever program seems suitable, to create stimuli as inputs to the DVS silicon retina. For static targets, we ensure the images flicker at a rate sufficient to drive the DVS. DVS output is directly fed into the SAC chip which provides an identification of the target position as a spike-based output. The SAC would alternate between competing stimuli if they have the same salience level.

A second channel subsamples the DVS input only to the local area indicated by the SAC. This subsample represents a 32x32 section of the complete 128x128 DVS image. Both the SAC output and the subsample are fed into the SpiNNaker chip (via a host system to handle the protocol translation between different spiking formats. The SpiNNaker chip then uses an internal network (which is a radically simplified approximation of the visual cortex and basal ganglia) to provide top-down feedbak to the SAC that biasses its selection of attention towards the preferred stimulus orientation.

A more advanced development will hook in a Bayesian predictor at the SpiNNaker outputs, that predicts the next likely position of the preferred stimulus based on relative motion and position of target and distractors. This network will only be connected in Phase 3.

 #!protected
 #:Access restricted to participants. Please log in.

Task allocation: After discussion, the workgroup agreed to the following division of labour

Daniel Sonnleithner and Francesco Galluppi will work on low-level SAC-SpiNNaker interfacing

Francesco Galluppi and (possibly) Sergio Davies will work on generating the subsampled DVS images

Mikhail Burtsev and Alex Rast will work on creating the SpiNNaker models.

Zeno Jonke will work on the creating the experimental environment and the Bayesian predictor network.

Progress to date:

Francesco and Daniel have resolved most of the hardware interfacing issues, by connecting through a host PC. The critical component is the SpikeServer system that supplies packet I/O to SpiNNaker. Daniel has the SAC successfully integrated to the DVS.

Mikhail Burtsev has generated a first model of the neural network for SpiNNaker, closely approximating (at a very simplified level) the model of Kevin Gurney for the basal ganglia.

Francesco and Alex have generated the SpiNNaker software subsystem necessary to support direct inputs to neurons via virtual synapses, thus bypassing the issue of having to instantiate internal neurons in SpiNNaker to perform this function.

The system as actually implemented incorporated all the planned elements of the attentional system except the SpiNNaker feedback to the SAC. Through the host PC the group was able to create an interface from the SAC to SpiNNaker, and to select the subregion from the retina. We then fed the subsample and direct SAC input to SpiNNaker to detect the region of attention.

The group created a series of stimuli ambiguous from the point of view of the SAC, in that they had identical salience, with patterns of horizontal and vertical lines in the stimulus images below the resolution of the SAC. Thus the chip would oscillate in attention between the two stimuli, as they competed for attention during presentation. However, we selected the horizontal stimulus as the preferred orientation direction for the SpiNNaker system. Using a neural network detector on SpiNNaker the system was able to identify the horizontally-striped stimulus as the focus for attention, and activate an output population when this stimulus was the selected attentional focus of the SAC, while remaining quiescent during attendance on the vertically-striped stimulus. Consistent firings of the output group indicate the probability that by implementing the control path to the SAC the SpiNNaker chip would be able consistently to direct the SAC's attention towards the horizontally-striped stimulus, by selective biassing of the inputs.

The system achieved a full integration of multiple spiking chips: the DVS silicon retina, the SAC chip, and SpiNNaker. Such a system definitely demonstrates the ability to connect AER-signalling devices into a complete cognitive-sensory pathway. More importantly, perhaps, the system shows the potential to develop a foveation system by using heterogeneous devices to control different parts of the task. The silicon retina is ideal for creating a high-resolution, high-response rate input sensor, but needs considerable back-end processing. The SAC chip provides a good model for how to achieve the first stages of the input processing pipeline, preprocessing input to higher cortical areas. By providing a relatively low-resolution but high-speed immediate identification of objects of interest in a scene, it can quickly direct the visual system towards the objects without having to do extensive scene analysis. This in turn makes it a good match for the SpiNNaker system that models cortical layers at higher levels of abstraction, where a more informative world view may be necessary. SpiNNaker is efficient at implementing complex areas providing, e.g. recognition of a specific high-resolution stimulus pattern, but without input preprocessing of the form the SAC provides, would have been ovewhelmed by the data-processing overhead. All of this is closely similar to the real structure of the visual system which provides hierarchies of visual scene processing not only in space but also in time.

Finally, such systems show how heterogenous combinations of chips can overcome the hardware limitations of any one component. For example, SpiNNaker would not be able in current form to implement the full 128*128-pixel detector within a single chip but the SAC can reduce the input space to a much more manageable 32*32 array. Likewise the SAC does not possess the high-scale resolution to detect fine structural details that may have an impact on attention, nor does it have the configurability to change the network architecture for more than a simple model of single-object attention. These are capabilities that SpiNNaker can easily provide. Overall, the project demonstrated an important direction for future research at CapoCaccia? and elsewhere: integration of multiple neuromorphic chips into complete systems that perform a defined and biologically relevant task.

Running models on spiNNaker using PyNN tutorial

Spiking neural network models can be implemented on the spinnaker without any knowledge about the hardware by using PyNN, a description language for spiking neural networks written in python.

For more information about PyNN visit  http://www.neuralensemble.org/trac/PyNN/, talk to me (francesco.galluppi@…) or follow the PyNN tutorial on the FACETS/BrainScaleS workgroup http://capocaccia.ethz.ch/capo/wiki/2011/bsshw11.

As a basic introduction you can download the synfire_example.py on this page http://capocaccia.ethz.ch/capo/attachment/wiki/2011/rta11/synfire_example.py.

You can also take a look at last year's wiki page which contains more examples on how to use PyNN and SpiNNaker http://capocaccia.ethz.ch/capo/wiki/2010/spinn10

See also