The discussion group will meet at 19:00 on Saturday 7th May in the main lecture room (Sala Rosa).
You can try a read-only instance of CATMAID with neuronal tissue here: http://incf.ini.uzh.ch/catmaid/
For the session this evening I'll try to set up a local copy that anyone can edit.
I started by presenting the data and the problem - showing our current interface, and the likely scale of the annotation task. (Each neuron takes between half a day and two days for identifying all synaptic partners and tracing the midline.)
We looked at the example of Galaxy Zoo http://www.galaxyzoo.org/ and in particular:
- Their simple introductory quiz with feedback on your classifications
- The live classification interface:
- Available straight from the front page with no registration
- You can classify galaxies anonymously, but with a login you can track your progress
- A lively community forum - people are actively interested in the galaxies and other objects they are classifying
We talked about some experiences with previous crowd-sourcing projects, e.g. video classification at http://www.theyworkforyou.com/video/ - a few users ended up doing a huge proportion of the classification.
A few of the things that we discussed are summarized below:
- Full 3D interfaces would require better registered data than we currently have, but would be an interesting possibility for the future.
- A point and click "shooting" interface where users have to try to hit as close to the centre of a membrane outline as they can (as quick as they can?)
- The simplest possible interfaces would have two images side-by-side, with the challenge to identify a corresponding midline point in the one on the right, based on the marked point in the left.
- The principle that users have to be encouraged to traverse neurons in both directions through the stack to deal with branches and merges.
- Training would need to include synapse identification. They should mark possible synapses in single slices, and afterwards when aggregating data we check for evidence in 3 consecutive slices.
- After sufficient users have redundantly traced starting from the same points, you can end up with a heatmap of likely midlines (possibly problematic is the requirement to cluster and aggregate these )
- These heatmaps could be used as part of the training exercise for new annotators - where would most experienced annotators click here?
- Key shortcuts are needed for everything to help the experienced annotators to go fast, and avoid frustration
- Additional keys to support optionally marking extra interesting features, e.g. mitochondria, dense-core vesicles, etc.
- There are problems with this interface where a neuron widens a great deal before branching - midpoints are no longer easy to define. (These cases are relatively rare, however, and the heatmap may just diffuse nicely.)
- Start points can be seeded from our many existing annotations, or an expert annotator could quickly go through marking every midpoint in a particular section.
- We think there is a community out there who are interested in seeing and exploring neurons at EM resolutions - perhaps not as obviously of widespread interest as galaxies (or space in general) but maintaining community interest would be key to the success of the project.
- Amazon Mechanical Turk https://www.mturk.com/mturk/welcome is an obvious option if one is happy to pay people to do the annotation - the same interface could be used, though.
