Statistical Choices Clause Samples

Statistical Choices. Five decisions relating to statistical choices were found to affect IAA. These choices are the population-sample ratio, method of selection, treatment of outliers, sample selection timing and granularity. Each of these choices are detailed in the following subsections. 1. Population-sample ratio In this study double annotators initially coded between 10% and 100% of each discipline within the corpus. The medical abstracts resulted in an exceptionally high degree of annotator agreement, while abstracts in information theory resulted in far less agreement. In disciplines that were easier to code, annotators coded more abstracts. Thus, reporting the true total number of double annotated abstracts regardless of discipline results in a higher IAA than reporting the IAA for 10% of each discipline.