Explain like I'm five
Imagine you're reading a sentence and you break it into pieces like 'I', 'love', 'pizza'. Each piece is a token. It's like cutting a sandwich into bite-sized pieces so you can eat it one piece at a time.
Artificial Intelligence Difficulty 35/100
Token
Small pieces, big meaning.
A token is the smallest unit of text that an AI model processes, like a word or part of a word.
Why it matters
Tokens are how AI understands and generates language, so they determine how much text you can input or output. You encounter tokens every time you use a chatbot or text generator—they limit how long your conversation can be.
Common misconception
Many think a token equals exactly one word, but it can be parts of words like 'un' in 'unhappy' or punctuation. This means a single word can become multiple tokens, affecting costs and limits.
Formal definition
In natural language processing, a token is a discrete unit of text, such as a word, subword, or character, produced by tokenization. Tokenization splits raw text into these units, which models then map to numerical representations for processing.