Common use of Annotation Guidelines Clause in Contracts

Annotation Guidelines. The annotation guidelines were created in an iterative process. A first draft was created, containing general guidelines as well as specific examples of difficult situations. Two archaeologists used the guidelines to annotate around 100 sen- tences, and these annotations were compared to our own desired annotations to see where problems and inconsistencies were encountered. This information was then used to update the guidelines, after which they were tested again. This led to an IAA (F1 score, further explained in section 3.5.1) of 0.94 between the two testers, which we consider sufficient for this task. During the annotation process itself, whenever one of the annotators ran into a situation that was unclear, this was added as an example to the guidelines. The annotation guidelines (in Dutch) can be downloaded as part of the data set (Brandsen, 2019).

Annotation Guidelines. The annotation guidelines were created in an iterative process. A first draft was created, containing general guidelines as well as specific examples of difficult situations. Two archaeologists used the guidelines to annotate around 100 sen- tences, and these annotations were compared to our own desired annotations to see where problems and inconsistencies were encountered. This information was then used to update the guidelines, after which they were tested again. This led to an IAA (F1 score, further explained in section 3.5.1) of 0.94 between the two testers, which we consider sufficient for this task. During the annotation process itself, whenever one of the annotators ran into a situation that was unclear, this was added as an example to the guidelines. The annotation guidelines (in Dutch) can be downloaded as part of the data set (Brandsen, 2019).. 40 CHAPTER 3. DATA SET