Agreement-based Learning. Xxxxx et al. [2006] first introduce agreement-based learning into word alignment: encouraging asymmetric IBM mod- els to agree on word alignment, which is a latent struc- ture in word-based translation models [Xxxxx et al., 1993]. This strategy significantly improves alignment quality across many languages. They extend this idea to deal with more latent-variable models in grammar induction and predicting missing nucleotides in DNA sequences [Xxxxx et al., 2007]. Xxx et al. [2015] propose generalized agreement for word alignment. The new general framework allows for arbitrary loss functions that measure the disagreement between asym- metric alignments. The loss functions can not only be defined between asymmetric alignments but also between alignments and other latent structures such as phrase segmentations. In attention-based NMT, word alignment is treated as a parametrized function instead of a latent variable. This makes word alignment differentiable, which is important for training attention-based NMT models. Although alignment matrices in attention-based NMT are in principle “symmetric” as they allow for many-to-many soft alignments, we find that unidi- rectional modeling can only capture partial aspects of struc- ture mapping. Our contribution is to adapt agreement-based learning into attentional NMT, which significantly improves both alignment and translation.
Agreement-based Learning. The key idea of agreement-based learning is to train a set of models jointly by encouraging them to agree on the hidden variables (Liang et al., 2006; Liang et al., 2008). This can also be seen as a particular form of posterior constraint or poste- rior regularization (Grac¸a et al., 2007; Ganchev et al., 2010). The agreement is prior knowledge and alignment and parsing (Xxxxxxx et al., 2010), tok- enization and translation (Xiao et al., 2010), pars- ing and translation (Liu and Liu, 2010), alignment and named entity recognition (Chen et al., 2010; Wang et al., 2013). Among them, Zhang et al. (2003)’s integrat- ed search algorithm for phrase segmentation and alignment is most close to our work. They use Point-wise Mutual Information to identify possi- ble phrase pairs. The major difference is we train models jointly instead of integrated decoding.
Agreement-based Learning. The basic idea of our work is to encourage the source-to-target and target-to-source translation models to agree on both phrase and word align- = argmax m T
Agreement-based Learning. Agreement-based learning has been proven as a useful paradigm in the language community (Xxxxx et al., 2006, 2007; Xxxxx et al., 2016). The core BlonDe Scores on the XXX dataset for X->En Directions Baseline Model BMA-SBT Averaged Score 50 52 54 56 58 60 → Figure 3: Averaged BlonDe scores from six directions in (X En) on the dataset of XXX Talks evaluated with BMA-SBT and the Baseline Model (Document-level). idea is to minimize the difference in the represen- tations between the inputs with the same meaning. Some multilingual pre-training methods such as Chi et al. (2021) are relevant to agreement-based learning in the way that they shrink the distance of cross-lingual representations between parallel data. Xxxxx et al. (2019) proposed to enforce an agreement on the output with left-to-right and right- to-left inputs on recurrent neural networks for ma- chine translation. Xxxx et al. (2020) proposed to use phrase-level agreement for machine translation. Still, Xxxx et al. (2021c) is the closest work to ours, which encourages agreement between par- allel data in different languages to have the same translation outputs. A very recent concurrent work uses MA to close the gap between source and tar- get languages (Gao et al., 2023). Our work creates synthetic data and employs bidirectional KL loss to enforce the multilingual agreement bidirectionally.
