Interpreting Interrater Agreement Sample Clauses

Interpreting Interrater Agreement. After calculating the value of Kappa, the next question is “how do we interpret it?” There are two general approaches for interpreting such measures. The first is with comparison to previously established baselines. However, given that there are no such baselines in software engineering, this approach is not feasible. The second approach is to establish some general benchmarks based on factors such as: what has been learned and accepted in other disciplines, experience within our own discipline, and our intuition. As a body of empirical knowledge is accumulated on software process assessments, we would evolve these benchmarks to take account of what has been learned. We resorted to guidelines developed and accepted within other disciplines. To this end, Xxxxxx and Xxxx [13] have presented a table that is useful and commonly applied for benchmarking the obtained values of Kappa. This is shown in Figure 7. Xxxxxxx [8] notes that while this table is arbitrary, it is still potentially useful for interpreting values of Kappa. In addition, we can test the hypothesis of whether the obtained value of Kappa meets a minimal requirement (following the procedure in [9]). The logic for a minimal benchmark requirement is that it should act as a good discriminator between assessments conducted with a reasonable amount of rigor and precision, and those where there was much misunderstanding and confusion about how to rate practices. It was thus deemed reasonable to require that agreement be at least moderate (i.e., Kappa > 0.4). Based on the results reported here and other studies already completed [5], this minimal value was perceived as a good discriminator. It should be cautioned, however, that the benchmark that we suggest above should only be considered initial. If, after further empirical study, it was found that this benchmark fails all SPICE processes, pass all of them, or pass ones that intuitively should be failed and vice versa, then the benchmark should be modified to strengthen or weaken the requirement. Kappa Statistic Strength of Agreement <0.00 Poor 0.00-0.20 Slight 0.21-0.40 Fair 0.41-0.60 Moderate 0.61-0.80 Substantial 0.81-1.00 Almost Perfect Figure 7: The interpretation of the values of Kappa.