A tokenizer is a algorithms that breaks text into smaller units – called tokens – that a language model like GPT can understand and process. These can be whole words, parts of words, or even individual characters. Tokenizing input (and output) is also necessary for the billing model for paid AI services.