Tokens, Context Windows, and Embeddings — Demystified

Tokens

Models don't read whole words — they read tokens, sub-word chunks produced by a tokenizer. 'Understanding' might become two tokens; a rare name might become four or five. This is why prices and limits are usually quoted per token, not per word.

Context windows

The context window is the maximum number of tokens a model can consider at once — prompt plus conversation history plus any retrieved documents. Exceed it and the oldest information simply falls out of view.

Embeddings

Embeddings are numerical vector representations of text that place similar meanings near each other in high-dimensional space, which is the mechanism behind semantic search and retrieval-augmented generation (RAG).