Sentence segmentation Clause Samples

Sentence segmentation. The next step in processing to be looked at is sentence segmentation. In order to assess the quality of some of the PANACEA tools in this area, a standard sentence segmentation module was used, namely the one delivered with the Europarl corpus (▇▇▇.▇▇▇▇▇▇.▇▇▇/▇▇▇▇▇▇▇▇ ). It was compared to one of the PANACEA sentence segmentisers, the LT-SSplit segmentiser (cf. D4.5). Both the German and the Italian documents were sentence-segmentised with both tools. The results are given in Table 4-1. No Segments it de Europarl 44.100 37.300 LT-SSplit 58.900 51.600 common 22.700 Table 4-1: Sentence Segmentation results It can be seen that the SSplit segmentiser produces significantly more sentences than the Europarl segmentiser; only about half of the sentences are identical. A closer look at the differences was taken as a consequence, by inspecting 1000 sentences in each language. The results of this evaluation are given in Table 4-2. Europarl correct 129 120 LT-SSplit correct 353 428 both wrong 518 452 The evaluation shows that the PANACEA tool is significantly more accurate than the Europarl segmentiser, in German more than in Italian. The rather high number of incorrect segmentations results mainly from two phenomena: • Treatment of enumerations as parts of a sentence, and not as a formal element3 • Interaction of sentence boundaries and paragraph boundaries. While LT-SSplit treats <p> … </p> markups also as sentence boundary, the Europarl segmentiser does not. In any case, the PANACEA sentence segmentation is clearly competitive in terms of industrial quality. The question is which effect the significant difference in sentence segmentation has on the alignment, as sentences form the basis of alignment. 3 This leads to mistakes when the numbers are treated as ordinals, and the constituents are moved in translation: (de) „2. ▇▇▇▇▇ sehen wir gern„ -> (en) „We love to see 2. Lions„ This is considered to be bad writing in handbooks for technical authors, but still occurs frequently in texts.