[ICML 2021 Oral] We show pure attention suffers rank collapse, and how different mechanisms combat it.
