Machine Translation. The ultimate goal of our system is to improve the agreement between the automatically generated parse trees and word alignments that are used as training data for syntactic ma- chine translation systems. Given the amount of variability between the outputs of different parsers and word aligners (or even the same systems with different settings), the best way to improve agreement is to learn a transformation sequence that is specifically tuned for the same annotators (parsers and word aligners) we are evaluating with. In particular, we found that though training on the English Chinese Translation Treebank produces clean, interpretable rules, preliminary experiments showed little to no improvement from using these rules for MT, primarily because actual alignments are not only noisier but also sys- tematically different from gold ones. Thus, all rules used for MT experiments were learned from automatically annotated text. As mentioned in Section 5.4, we could, in principle train on all 506k sentences of our MT training data. However, this would be quite slow: each iteration of the training proce- dure requires iterating through all n training sentences7 once for each of the m candidate transformations, for a total cost of O(nm) where m grows (albeit sublinearly) with n. Be- cause the agreement score metric is additive, the most useful transformations (and hence the first ones found by the greedy learning procedure) are typically the ones that are trig- gered the most frequently, so any reasonably sized training set is likely to contain them. Thus, it is not actually likely that dramatically increasing the size of the training set will yield particularly large gains. To train our TBL system, we therefore extracted a random subset of 3000 sentences to serve as a training set.8 We also extracted an additional 1000 sentence test set to use for rapidly evaluating agreement score generalization. Figure 5.10 illustrates the improvements in agreement score for the automatically annotated data, analogous to Figure 5.8. The same general patterns hold, although we do see that the automatically annotated data is more idiosyncratic and so more than twice as many transformations are learned before training set agreement stops improving, even though the training set sizes are roughly the same.9 Furthermore, test set generalization in the automatic annotation setting is a little bit worse, with later transformations tending to actually hurt test set agreement. Our MT setup ...
