Translations:Attention Mechanisms/17/en
The scaling factor $ \sqrt{d_k} $ prevents the dot products from growing large in magnitude as the key dimension $ d_k $ increases, which would push the softmax into regions of extremely small gradients.
The scaling factor $ \sqrt{d_k} $ prevents the dot products from growing large in magnitude as the key dimension $ d_k $ increases, which would push the softmax into regions of extremely small gradients.