What is Token?
Token
A. What is Token
The smallest unit an AI uses to process text. Pricing and the maximum input length (context) are both measured in tokens.
A token is the smallest unit an LLM uses to handle text. In English a token is often a word or part of a word; in Japanese it tends to be a character or fragment of a word. As a rough guide, Japanese consumes roughly one token per character.
API pricing is usually charged by the volume of input tokens plus output tokens, and the maximum length of text you can input (the context window) is also expressed in tokens. The longer the document you process, the more tokens you consume.