📝 主旨内容

为了让信息不被“淹没”

与Softmax共享有关

Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension 𝑑model. We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [24]. In the embedding layers, we multiply those weights by √𝑑model

与Embedding的初始化有关

embedding的初始化采用了Xavier初始化，它的方差是1/sqrt（dk），所以这样做可以把方差恢复成1。

weight (Tensor) – the learnable weights of the module of shape (num_embeddings, embedding_dim) initialized from N(0,1)

词嵌入的初始化均值为 0.0，标准差为 embedding_dim ** -0.5

为了让训练更加稳定

🤗 总结归纳

📎 参考文章

• https://stackoverflow.com/questions/56930821/why-does-embedding-vector-multiplied-by-a-constant-in-transformer-model

• https://datascience.stackexchange.com/questions/87906/transformer-model-why-are-word-embeddings-scaled-before-adding-positional-encod

• https://github.com/pytorch/tutorials/issues/2849

• https://discuss.pytorch.org/t/pytorch-transformers/76993/2

• https://www.reddit.com/r/MachineLearning/comments/fbn0oe/d_attention_is_all_you_need_transformer_decoder/

• https://docs.pytorch.org/docs/stable/generated/torch.nn.Embedding.html

• https://github.com/espnet/espnet/issues/2797

• https://www.zhihu.com/question/12138033385/answer/116327921219