Where Transformer uses Mixture of Experts
In MoE LLM, where is Mixture of Experts usually located: in which layer of Transformer and why is it done?
Answer as you would in an interview
Answer aloud or write down a few key points. The text field is optional; the review opens after your attempt.