Routing-by-agreement and Self-Attention Layer Sample Clauses

Routing-by-agreement and Self-Attention Layer. In this section, we want to analyse the similarities be- tween the two mechanisms. The routing-by-agreement and the self-attention layer are built on the same idea of ex- tracting a probability distribution that describes the relation among the lower layer and the upper layer. In capsule net- works, the dynamic routing determines the connection from the lower layer to the higher layer of capsules, in par with this the self-attention layer decides how to attend to differ- ent parts of the input and how information from different parts contribute to the updates of representations. It is pos- sible to map the attention weights the self-attention layer to assignment probabilities in capsule net. The probabilities distribution differs between the two mechanisms: in capsule nets, the assignment probabilities are computed bottom-up, in self-attention the attention is computed top-down. i.e. the attention weights in self-attention are distributed over the representations in the lower layer, but in capsule nets, the assignment probabilities are distributed over the higher layer capsules. The attention probabilities are computed based on the similarity of the representations in the same layer, but this is equivalent to the assumption that the higher layer is first initialised with the representations from the lower layer and then it is updated based on the attention probabilities computed by comparing these initial represen- tations with the representations from the lower layer.