Query-Key Normalization Improves Low-Resource Transformer Translation

This episode examines Query-Key Normalization for Transformers, which swaps the attention dot product for a scaled cosine similarity. It reports an average gain of 0.928 BLEU across five low-resource translation pairs, using small models and corpora of roughly 10,000 to 214,000 sentence pairs. The discussion explains why unbounded dot products can push softmax toward near one-hot attention, and how L2-normalizing queries and keys bounds the scores. A learnable scale, initialized by a rule of thumb based on sequence length, then restores attention sharpness. It also places the method against the ScaleNorm, FixNorm and PreNorm baseline, and flags a confound: the baseline uses ScaleNorm while the QKNorm model uses LayerNorm. Listeners get a careful look at how much of the gain the change itself can claim.

Sources:

1. Query-Key Normalization for Transformers — Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, Yuxuan Chen, 2020

http://arxiv.org/abs/2010.04245

2. Transformers without Tears: Improving the Normalization of Self-Attention — Toan Q. Nguyen, Julian Salazar, 2019

https://scholar.google.com/scholar?q=Transformers+without+Tears%3A+Improving+the+Normalization+of+Self-Attention

3. Query-Key Normalization for Transformers — Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, Yuxuan Chen, 2020

https://scholar.google.com/scholar?q=Query-Key+Normalization+for+Transformers

4. Scaling Vision Transformers to 22 Billion Parameters — Mostafa Dehghani, Josip Djolonga, Basil Mustafa, et al. (Google Research), 2023

https://scholar.google.com/scholar?q=Scaling+Vision+Transformers+to+22+Billion+Parameters

5. Small-scale proxies for large-scale Transformer training instabilities — Mitchell Wortsman, Peter J. Liu, Lechao Xiao, et al., 2023

https://scholar.google.com/scholar?q=Small-scale+proxies+for+large-scale+Transformer+training+instabilities