Tokenisation
Tokenisation is the process of splitting text into tokens that a language model can process.
What is Tokenisation?
Most modern models use subword methods such as byte-pair encoding, which keep common words whole and split rare words into pieces. Tokenisers can be less efficient for some languages, making them costlier to process.
How it works
The tokeniser applies a learned vocabulary of subword pieces to break text into the fewest suitable tokens.
← All Artificial Intelligence & Superintelligence definitions · ← Back to the A–Z definitions