Inference & performance · Concept

TPS

Also called: tokens per second

Tokens per second; specify whether it measures one request or aggregate system throughput.

Related terms

TTFT · ITL · TPOT

Primary references

Reference definition