Analysis of Alignment Matrices Sample Clauses

Analysis of Alignment Matrices. We observe that a target word is prone to connect to too many source words in the alignment matrices produced by indepen- dent training. For example, in the lower alignment matrix of Figure 2(a), the third Chinese word “buxi” is aligned to three English words: “president”, “xxxx”, and “condemns”. In ad- dition, all the three alignment probabilities are relatively low. Similarly, four English words contribute to generating the last Chinese word “gongji”: “condemns”, “suicide”, “boming”, and “attack”. In contrast, agreement-based joint training leads to more concentrated alignment distributions. For example, in the lower alignment matrix of Figure 2(b), the third Chinese word “buxi” is most likely to be aligned to “xxxx”. Likewise, the attention to the last Chinese word “gongji” now mainly fo- cuses on “attack”. To measure the degree of concentration of attention, we define the attention entropy of a target word in a sentence pair as follows: Σ Hyn = − A(θ)n,m log A(θ)n,m (14) s=1 m=1 is calculated at the word level rather than character level.

Analysis of Alignment Matrices. We observe that a target word is prone to connect to too many source words in the alignment matrices produced by indepen- dent training. For example, in the lower alignment matrix of Figure 2(a), the third Chinese word “buxi” is aligned to three English words: “president”, “xxxx”, and “condemns”. In ad- dition, all the three alignment probabilities are relatively low. Similarly, four English words contribute to generating the last Chinese word “gongji”: “condemns”, “suicide”, “boming”, and “attack”. In contrast, agreement-based joint training leads to more concentrated alignment distributions. For example, in the lower alignment matrix of Figure 2(b), the third Chinese word “buxi” is most likely to be aligned to “xxxx”. Likewise, the attention to the last Chinese word “gongji” now mainly fo- cuses on “attack”. To measure the degree of concentration of attention, we define the attention entropy of a target word in a sentence pair as follows: Σ Hyn = − A(θ)n,m log A(θ)n,m (14) s=1 m=1 is calculated at the word level rather than character level. Given a parallel corpus D = {⟨x(s), y(s)⟩}S , the aver- System Training Direction Dev. Test Moses Indep. E→F F→E 28.38 28.52 32.31 30.93 RNNsearch Indep. E→F F→E 29.06 28.32 32.69 29.99 Joint E→F F→E 29.86∗∗++ 29.01∗∗++ 33.45∗∗++ 31.51∗∗++ Table 5: Results on the English-French translation task. The BLEU scores are case-insensitive. “**”: significantly better than XXXXX (p < 0.01); “++”: significantly better than RNNSEARCH with independent training (p < 0.01). age attention entropy is defined as After analyzing the alignment matrices generated by RNNSEARCH [Xxxxxxxx et al., 2015], we find that model- S N y y c(y, D) n