Token
In language models, a token is the basic unit of text the model reads and writes, typically a word, part of a word or a punctuation mark.
What is Token?
In English, one token is roughly three-quarters of a word on average. Model limits, speed and pricing are usually expressed in tokens, for example the price per million tokens or a 200,000-token context window.
How it works
A tokeniser splits text into tokens and maps each to an ID number, which the model then converts into an embedding.
← All Artificial Intelligence & Superintelligence definitions · ← Back to the A–Z definitions