Common Contracts
1 similar null contracts
Anchoring and Agreement in Syntactic AnnotationsSeptember 3rd, 2018
We present a study on two key character- istics of human syntactic annotations: an- choring and agreement. Anchoring is a well known cognitive bias in human decision mak- ing, where judgments are drawn towards pre- existing values. We study the influence of anchoring on a standard approach to creation of syntactic resources where syntactic annota- tions are obtained via human editing of tagger and parser output. Our experiments demon- strate a clear anchoring effect and reveal un- wanted consequences, including overestima- tion of parsing performance and lower qual- ity of annotations in comparison with human- based annotations. Using sentences from the Penn Treebank WSJ, we also report systemat- ically obtained inter-annotator agreement es- timates for English dependency parsing. Our agreement results control for parser bias, and are consequential in that they are on par with state of the art parsing performance for En- glish newswire. We discuss the impact of our findings on strategi
