Inter-Annotator Agreement. ON ANNOTATION EFFORT OF XXXX ET AL. (2003) Xxxx et al. (2003) used Xxxxx et al.’s (1999) kappa statistic methodologies to measure various aspects of the inter-annotator agreement on their RST based corpus. Five topics were presented to fully cover the typical agreement issue of those kinds of corpora. The first topic deals with unit segmentation and the rest of them suggest methodologies for the issues emerging with the hierarchical structure of the corpora. Essentially, in all the methodologies for hierarchical aspects, hierarchical structure was flattened to a linear table by considering each possible segment pairs as units which constitute the source data to compute the kappa statistic. The following is a suitable example, which is a modified portion of a sample annotation from the study of Xxxxx et al. (1999), to clarify the claim above. In Figure 4, there are two nuclearity segmentation examples for two levels that represent two hierarchical discourse structures of the same text: Segmentation 1 N N S N S 1 0 N S Segmentation 2 Figure 4 Two sample hierarchical RST discourse structures for the same text. (N=Nucleus, S=Satellite) As a result of flattening, the following data table is constructed from the discourse structure above: Table 5 Data table of Figure 4 Segment Segmentation 1 Segmentation 2 [0,0] none N [0,1] N N [0,2] N None [1,1] none S [1,2] none None [2,2] S S The constructed agreement table is used as the input to the kappa statistic. For this sample the attributes of the kappa statistic are 2 annotators (Segmentation 1, Segmentation 2), 3 categories (N, S, none), and 9 samples (segment pairs). In the light of this explanation, five inter-annotator agreement aspects are as follows:
Inter-Annotator Agreement. For most tasks, Xxxxx’x Kappa is reported as a measure of IAA, and is consid- ered the standard measure (XxXxxx, 2012). But for Named Entity Recognition, Kappa is not the most relevant measure, as noted in multiple studies (Xxxxxxxx & Xxxxxxxxxx, 2005; Xxxxxx et al., 2011). This is because Kappa needs the num- ber of negative cases, which isn’t known for named entities. There is no known number of items to consider when annotating entities, as they are a sequence of tokens. A solution is to calculate the Kappa on the token level, but this has two associated problems. Firstly, annotators do not annotate words individually, but look at sequences of one or more tokens, so this method does not reflect the annotation task very well. Secondly, the data is extremely unbalanced, with the un-annotated tokens (labelled "O") vastly outnumbering the actual entities, un- fairly increasing the Kappa score. A solution is to only calculate the Kappa for tokens where at least one annotator has made an annotation, but this tends to underestimate the IAA. Because of these issues, the pairwise F1 score calculated without the O label is usually seen as a better measure for IAA in Named Entity Xxxxx’x Kappa on all tokens 0.82 Xxxxx’x Kappa on annotated tokens only 0.67 F1 score 0.95 Table 3.4: Inter-annotator agreement measures on 100 sentence test document. Calculated by doing pairwise comparisons between all combinations of annotators and averaging the results. Recognition (Xxxxxxx et al., 2012). However, as the token level Kappa scores can also provide some insight, we provide all three measures but focus on the F1 score. The scores are provided in Table 3.4. These scores are calculated by averaging the results of pairwise comparisons across all annotators. We also cal- culated these scores by comparing all the annotators against the annotations we did ourselves, and obtained the same F1 score and slightly lower Kappa (-0.02).
Inter-Annotator Agreement. The need to ascertain the agreement and reliabil- ity between coders for segmentation was recognized − 3Georgescul et al. (2006, p. 48) note that both FPs and FNs are weighted by 1/N−k, and although there are “equiprobable possibilities to have a [FP] in an interval of k units”, “the total number of equiprobable possibilities to have a [FN] in an inter- val of k units is smaller than (N k)”, making the interpretation of a full miss as a FN less probable than as a FP. by Passonneau and Xxxxxx (1993), who adapted the percentage agreement metric by Xxxx et al. (1992,
Inter-Annotator Agreement. Similarity alone is not a sufficiently insightful mea- sure of reliability, or agreement, between coders.
