Inference & performance · Concept

Batch inference

Process a collection of requests as an offline or scheduled workload.

Related terms

TTFT · ITL · TPOT

Primary references

Reference definition