Evaluating Interrater Agreement Sample Clauses

Evaluating Interrater Agreement. To evaluate interrater agreement3, we treat the SPICE adequacy ratings as being on a nominal scale. ▇▇▇▇▇ [2] has defined coefficient Kappa (κ) as an index of agreement that takes into account agreement that could have occured by chance. The value of Kappa is the ratio of observed excess over chance agreement to the maximum possible excess over chance agreement. See [5] for the details of calculating Kappa. If there is complete agreement, then κ=1. If observed agreement is greater than chance, then κ>0. If observed agreement is less than would be expected by chance, then κ<0. The minimum value of κ depends upon the marginal proportions. However, since we are interested in evaluating agreement, the lower limit of κ is not of interest. The value of Kappa depends strongly on the marginal distributions (see [1]). This means that the same rating procedure can potentially produce different values of Kappa depending on the proportion of each of the adequacy levels that were rated for a given process. However, ▇▇▇▇▇ does have the advantage of taking into consideration chance agreement. In addition, when compared to perhaps more intuitive indices of agreement such as percentage agreement, Kappa tends to have lower values than percentage agreement [12]. Therefore, ▇▇▇▇▇ is more conservative. It is then noted in [12] that “The tradition in science to accept conservative rather than liberal estimates suggests that percentage agreement is the least desirable [when compared to other reliability estimates, including Kappa]”. Therefore, we have a strong justification for using the coefficient Kappa over percentage agreement. 3 It should be noted that “agreement” is different from “association”. For the ratings from two teams to agree, the ratings must fall in the same adequacy category. For the ratings from two teams to be associated. it is only necessary to be able to predict the adequacy category of one team from the adequacy category of the other team. Thus, strong agreement requires strong association, but strong association can exist without strong agreement. For instance, the ratings can be strongly associated and also show strong disagreement. Develop System Requirements and Design (ENG.1) Specify System Requirements and Design Describe System Architecture Allocate Requirements Determine Release Strategy Develop Software Requirements (ENG.2) Determine Software Requirements Analyze Software Requirements Determine Operating Environment Impact Evaluate Requirements wi...