Continuous batching in LLM inference
What is continuous batching and why is it needed in the inference of large language models?
Answer as you would in an interview
Answer aloud or write down a few key points. The text field is optional; the review opens after your attempt.