Event based sensory processing

Members: Andreas Grübl, Angel Jimenez, Adrian Whatley, Andrew Dankers, Aurel A. Lazar, Bernabe Linares-Barranco, Mikhail Burtsev, Charles Clercq, Christoph Krautz, Christoph Posch, Daniel Sonnleithner, Dylan Richard Muir, Emre Neftci, Eric Müller, Daniel B. Fasnacht, Florian Jug, Jan Funke, Gabriele Spina, Hsin Chen, Hirotsugu Okuno, Eric Ryu, Jayawan Wijekoon, Jorg Conradt, Jonathan Tapson, Antoine Joubert, Kaijun Yi, Alexander Kazeka, Leslie Smith, Manuel J. Dominguez Morales, Mark Longair, Mohammad Abdollahi, Bernhard Nessler, Michael Pfeiffer, Rainer Beutelmann, Alexander Rast, Rodolphe Héliot, Ryad Benosman, Sadique Sheik, Yulia Sandamirskaya, Sergio Davies, Simon Friedmann, Shih-Chii Liu, Siohoi Ieng, Sebastian Jeltsch, Stefan Habenschuss, Sushmita Allam, Tobi Delbruck, Thomas Pfeil, Tobias Pietzsch, Tao Zhou, Vasilis Thanasoulis, Victor Benichoux, Viviane Ghaderi, Carlos Zamarreno-Ramos

Leaders: Tobi Delbruck, Shih-Chii Liu, Christoph Posch, Ryad Benosman, Jorg Conradt, Matthew Cook, Teresa Serrano Gotarredona, Bernabe Linares-Barranco

We are interested in hardware and algorithms that use the notions of events that combine event-based (AER) neuromorphic hardware with conventional digital post-processing. We'll build real-time hardware robotic systems and discuss silicon circuits and architectures, along with event-based digital processing algorithms. We'll use the  DVS silicon retina, the ATIS, and the  AER-EAR silicon cochlea in these projects.

We will probably use jAER for most of these projects and will run a jAER tutorial (sign up here) for new users. See 2011/jaer11 for downloads of jAER, the Java Development Kit, and the netbeans IDE.

This workgroup is closely related to 2011/spikingrobots11.

Discussions

We're interested in a variety of topics spanning transistor level implementations of event-based sensor circuits to algorithms for processing the sensor outputs. We hope to engage in ongoing debates about alternative approaches and new ideas for implementation.

Projects

Enter ideas here for projects....

Gesture tracking - combining DVS with Kinect

We will bring  DVS silicon retinas and a Microsoft Kinect sensor to CC and see whether the sum can be greater than the individual parts. The hope is that the Kinect can provide structured-light 3d information and the DVS can fill in the time domain.

See also the Kinect workgroup.

Labyrinth

The  labyrinth game (pictured below) takes hundreds of hours of human effort to master yet is controlled using just two parameters (the tilt of the table). It requires fast and precise visual reactions and learned control to navigate the steel ball bearing through the treacherous maze. The plan is to build a robotic labyrinth player based around this  Brio labyrinth, which has 3 different mazes of increasing difficulty. We'll use the  DVS silicon retina or the ATIS, combined with highly developed trackers in  jAER to track the ball, and we already have a handy servo controller board interfaced to jAER that we will use to control servos to tilt the table. The video below shows the setup. Some  Danish students already did a lovely job on robotic Labyrinth and we will try to outdo them.

The labyrinth game using the PID controller and cluster tracker is shown in the next video

Saccadic eye / ear

We recently built a very fast pan-tilt  DVS silicon retina (see video below) and also  a new AER-AER silicon cochlea, with which we can study practical real-time implementations of audio-visual sensor fusion and the roles of saccades and microsaccades and the spike events they produce in the eye, in early vision, as considered by  Michele Rucci. Perhaps we could follow a bouncing ball, using the pan-tilt to track the ball and the cochlea to hear the sound of bouncing, which will help the predictive tracking.

Multimodal integration

We will study how to integrate information from both the DVS silicon retina and the AER-EAR2 silicon cochlea to bimodal stimuli in the presence of either visual or auditory distractors.

Generalized object tracking

The simple rectangular object tracker in jAER has been useful in a number of demonstrator robots. The line tracker in the pencil balancer was a good example of an effective event-based tracker for an extended object. (See  pencil balancer,  robo goalie, and  laser tracker] for examples.) Now Matt Cook's so-called PigTracker has demonstrated how these ideas might be generalized. However, this basic algorithm needs quite a bit of refinement for actual usefulness. For example, it presently has no measure of its own confidence in its reasonableness, and often goes unstable. We aim to improve the algorithm and its implementation and use it in some kind of simple robotic demonstrator.

DVS Calibration - What does each pixel mean?

Introduction

For any fabricated chip, there will be mismatching among identically designed components. The DVS also has the same problem. This means that for the same light intensity change, each pixel of DVS will give out different number of events. In this sense, statistically, the events generated from each pixel have different meanings.

Project Goal

In this project, we try to use a simple statistical method to obtain a set of weights for a certain DVS chip. If the weight of a certain pixel is large, it means that it fires event less frequently and that each event from that pixel means more or it need a larger light intensity change to trigger an event. On the other hand, if the weight of a certain pixel is small, it means that the pixel fires a lot so that each event from that pixel should mean less or a smaller light intensity change could be enough to trigger the event. So if we could obtain such a set of weights, we could interpret the statistics of the events better. Method

We let the DVS observe the environment for a relatively long time and collect the events data obtained during the observation. As mentioned before, we want to have a better statistical interpretation of the DVS events. The important conjecture here is that with the continuous environment and continuous movement of the DVS, the total number of events within a neighborhood should be statistically equal. We don’t want to add the constraint over large area since that may be biased by the background itself. For example, the sky may fire more events than the ground in general. So the way to get out the statistics is as the following.

1. Go through all possible 8 by 8 neighborhoods within a image.

2. For each neighborhood, calculate the mean events number.

3. Go through all the pixels within this neighborhood and add the difference between the number of events of that pixel and the mean events number of this neighborhood to that pixel’s weight.

4. Normalize the weight into proper range, for example, from 0.5 to 1.5.

The detailed update rule could be found in the comment of the code. The code is under the package ch.unizh.ini.jaer.projects.thresholdlearner of jAER project source. The two matlab code files are written by me and is used to obtain the following result.

Result

Positive Events Weight

Negative Events Weight

Discussion

We've changed the neighborhood size and the amount of data we used to extract the statistics. But the result we could get is more or less the same. This means that using this method, we could indeed capture some of the nature of the difference between DVS pixels. But the problem is how to use this statistical result into real application. Obviously, it is not very good to apply this weight to each event directly. I think if there is some statistical application of the events, we could use this set of weight. And I'm happy to see whether this result could be really applied to improve the interpretation of the DVS events.

See also

See wiki:2011/jaer11 for downloads of Java Development Kit, netbeans, etc.