Full definition

Process of splitting text into smaller units (tokens) that the model can process, such as words or subwords.

Example in a business context

The word 'tokenization' might split into ['token', 'ization'].