Models, reasoning & training · Concept

Tokenization

Convert input into units from a model-specific vocabulary.

Related terms

Pretraining · Post-training · Base model

Primary references

Reference definition