Inference & performance · Concept

Continuous batching

Dynamically batch requests as they enter and leave inference execution.

Related terms

TTFT · ITL · TPOT

Primary references

Reference definition