UNMT Settings. In the training process for UNMT, we used the transformer-based UNMT toolkit5 and the settings of Xxxxxx et al. (2018b). That is, we used four 3xxxxx://xxxxxx.xxx/facebookresearch/ fastText 4xxxxx://xxxxxx.xxx/artetxem/vecmap 5xxxxx://xxxxxx.xxx/facebookresearch/ UnsupervisedMT layers in both the encoder and the decoder. Three out of the four encoder and decoder layers were shared between the source and target languages. The dimension of the hidden layers was set to 512. Training used a batch-size of 32 and the Xxxx optimizer (Xxxxxx and Ba, 2015) with an initial learning rate of 0.0001, β1 = 0.5. The vocabulary size was set to 60k by concatenating the source and target corpora. We performed 140 epochs6 (approximately 500K iterations) to train every model. The case-sensitive BLEU score computed with the multi bleu.perl script from Moses7 was used as the evaluation metric. For model selection, we followed the strategy described by Xxxxxx et al. (2018a). That is, the BLEU score computed between the original source sentences − 6The definition of epoch in UNMT is different from that in NMT. We followed the settings in Xxxxxx et al. (2018b)’s toolkit, i.e., 3500 iterations as one epoch. 7xxxxx://xxxxxx.xxx/moses-smt/ mosesdecoder and their reconstructions was used as the criterion. We selected the model that had the highest average BLEU score over the two translation directions. For the proposed methods, both UBWE agreement regularization and UBWE adversarial training were added as objective functions at the beginning of UNMT training. The detailed parameter settings are discussed in Section 6.
