Review Service API

Use Cases

1. simple identification of differences between two annotations
2. review of differences be filtered by the type of change
   • e.g. for treebanking, identify places where we changed tokens (and / or word ids) while ignoring changes to things like pos and head, and vice-versa
   • for alignments, maybe identifying a subset of a sentence to focus on
3. comparison against a single "gold standard"
4. comparison against a set of alternate versions
   • e.g. as in the case of comparison of different students work against each other
   • or against multiple acceptable interpretations
5. comparison across against similar constructions/usages in different sentences?

API Considerations

Inputs

Formats: XML and JSON Raw Data: both URIs and Text

• annotation+ for grading
• gold standard+ for comparison
• filtering/focus requirements
• customizations (e.g. labels)
  • need to put some thought into where the mapping of different tag sets takes place - i.e. on the client to the comparison service or by the comparison service

Outputs

Formats: XML and JSON

• itemization of differences/errors

Need to explore exact format of these outputs. One option is to add annotations inline in the input data that identify the errors. Might not be generalizable enough for different purposes. Also should consider what something like an LMS might need in order to use the data - for example to create customized assessments based on the error details, or to modify/customize lesson plans, etc.

Would be worth looking at some of the existing standards for this and what the different learning management systems support. Blackboard is closed so we shouldn't worry about that -- Moodle, Sakai, edX could be considered. Good resource to start with for LMS standards is IMS Global/QTI.

Note that that output data should take into consideration the type of information required for assessment down the line. This could include things like categorization of types of errors, identification of unique errors vs. repeated errors, level of granularity of the error (e.g. in the case of hierarchical tags), etc. See for examples Assessment Service Requirements.

Also worth looking at some existing scoring algorithms:

Perseus Hopper Scoring Code

1. IA: percentage of sentences with all relationships and head identifications made correction (with morphology excluded)
2. MorphScore: ( number of individual morph characteristics correctly identified) / (number of words * 9)
3. TreeScore: (( number of relationship tags correctly identified) + ( number of head words correctly identified ) ) / (number of words * 2)
   • in the calculation of number of relationship tags correctly identified, deducted only once for each unique misidentification in any given sentence

Consumers

• Perseids
• eLeaning tools
• scholars (inter annotator agreement)
• assessment services
• ...

Questions/Answers

1. is a comparison service the same as a grading service?
   • GH: I think no. They might more often than not be used in conjunction, but grading is a layer on top, that works with output of a comparison. Also adds flexibility to use different grading services (from simple scoring to more NLP-guided tools, that try to make some sense out of differences [highly language specific])
2. should service accept both text and uris in input or just uris?
   • GH: Both, to compare unsaved changes on the fly during an annotation sequence.
3. what role do tag customizations (i.e. for treebanking play)? For example, if comparing against a set of annotations, do we need to be able to restrict comparison to annotations that used the same tag set?
   • GH: As noted in Use Cases 2) we can always review selected things only, in such a case head and POS only. One could argue that heads are more important anyway. - Labels could still be compared if the customizations are mapped back to some sort of standard (the Prague label set e.g.) - Hint for the future: Giuseppe's suggestions about creating multi-layered label sets comes also into play here. Very short explanation: He suggests e.g. a genetive proper label, that contains another layer for more fine-grained genetive analysis (e.g. a possessivus or qualitatis). If the gold standard has gen.qual. and the annotation under review has only gen.proper we have another kind of error as when the annotation had SBJ or something like that.

Brainstorming

>>> START:AT

Here's my take on the task at hand. We should focus on first extracting metrics from a treebank, aka building a "report".

Report > Comparison > Grade

I don't have a strong linguistic background so I'm sorry if I'm abusing the vocabulary.

Two kinds of comparison.

Visual comparison.

• Possible solution.
  • Rendered tree images can be placed one on top of the other in semitransparent layers.
  • Structural differences can be determined instantly this way.

Computational comparison.

Question: What is a comparison?

2 reports smooshed into 1 designed to highlight differences.

Question: What is a report?

A collection of metrics so...
We have to define metrics.

A treebank file has two basic kinds of data.

1. Morphological identification
   • at the "word level"
     • morphological ids
   • at the "sentence level"
     • morphological id frequencies
     • morphological id positions
2. Syntactical structure
   • branching structure
     • branch count
     • branch locations
     • maximum depth
   • branch syntax-ids
     • branch id count
     • branch id locations

Once we know what metrics will be in our report we have to write...

1. Algorithms to extract the metrics.
2. Data structures to store them.

and then...

1. Algorithms to compare two sets of metrics.

and finally...

1. Algorithms to weigh the importance of metrics.

In short I think defining a report is our first step.

Report > Comparison > Grade

>>> END:AT