Attention difficulty, GQA, MQA and sliding window
How does scalable scalar attention work, what is its computational complexity, and what changes GQA, MQA, and the limited attention window?
Answer as you would in an interview
Answer aloud or write down a few key points. The text field is optional; the review opens after your attempt.