Adaptive GPU routing for LLM serving
There are several GPU workers in LLM serving. Why naive round-robin routing may be ineffective, and how to design an adaptive routing layer that takes into account GPU headroom and KV-cache reuse?
Answer as you would in an interview
Answer aloud or write down a few key points. The text field is optional; the review opens after your attempt.