Token
トークン(トークン)
BeginnerCore Concepts
A small unit of text (a word, part of a word, or punctuation) that AI models process. Models read and generate text as sequences of tokens.
Why It Matters
Token limits determine how much text a model can process at once, and API pricing is based on token usage.
Example in Practice
The sentence 'I love AI' might be split into 3 tokens: 'I', ' love', ' AI'.