Other Experimental Details Sample Clauses

Other Experimental Details. The BERTbase model and the RoBERTaBASE model use the same configuration. The two models both have 12 hidden transformer layers and 12 attention heads. The hidden size of the model is 768 and the intermediate size in the transformer layers is 3,072. The activation function in the transformer layers is gelu. Pre-training The batch size of 32 sequences is used for pre-training. Xxxx with the learning rate of 5 · 10−5, β1 = 0.9, β2 = 0.999, the L2 weight decay of 0.01, the learning rate warm up over the first 10% steps, and the linear decay of the 2Question type results for all models are in Appendices. learning rate are used. A dropout probability of 0.1 is applied to all layers. The cross-entropy is used for the training loss of each task. For the masked language modeling tasks, the model is trained until the perplexity stops decreasing on the development set. For the other pre-training tasks, the model is trained until both the loss and the accuracy stop decreasing on the development set. Fine-tuning For fine-tuning, the batch size and the optimization approach are the same as the pre-training. The dropout probability is always kept at 0.1. The training loss is the sum of the cross-entropy of two fine-tuning tasks as in section 3.3.