Utterance Order Prediction Sample Clauses

Utterance Order Prediction i The embedding ec from the utterance-level MLM (u-MLM) learns contents within Ui, but not across other utterances. In dialogue, it is often the case that a context is completed by multiple utterances; thus, learning attentions among the utterances is necessary. To create embeddings that contain cross-utterance features, the utterance order prediction model is trained (Figure 3.1(c)). Let D = D1 ⊕ D2 where D1 and D2 comprise the first and the second halves of the utterances in D, respectively. Also, let Dj = D1 ⊕ D2j where D2j contains the same set of utterances as D2 although the ordering may be different. The task is to determine whether or not Dj preserves the same order of utterances as D. i1 in For each Ui ∈ Dj, the input Ii = {CLSi} ⊕ Ui is created and fed into TE, already trained by u-MLM, to create the embeddings Ei = {ec, es, ew , .., ew }. The n sequence Ec = {ec, . . . , ec } is fed into two transformer layers, TL1 and TL2, that n generate the new utterance embedding list Tc = {tc, . . . , tc }. Finally, Tc is fed into a softmax layer that generates oν ∈ R2 to predict whether or not Dj is in order.