Model serving interview questions
Model serving
Why is serving a model different from serving an ordinary web request? (Easy)
What is dynamic batching and what does it trade? (Medium)
How do you handle cold starts when autoscaling model servers? (Hard)
p50 latency is fine and p99 is ten times worse. What are the likely causes in a model server? (Extra hard)