Tokenization, Context Windows & High-Dimensional Embeddings

Byte-Pair Encoding (BPE), SentencePiece, vector representations, and semantic distance metrics.

Language models do not read text directly—they process discrete numerical identifiers called tokens. In this lesson, we trace how strings convert into token embeddings in high-dimensional vector spaces (e.g. 1536 dimensions for OpenAI text-embedding-3-small).

Cosine Similarity

To measure semantic similarity between two normalized embeddings $\mathbf{u}$ and $\mathbf{v}$ :

$\text{Cosine Similarity} = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}$

LLMs & RAG: Interactive Lab

LLMs & RAGMatched to lesson

Simulates tokenization, subword breakdown, and dense embedding vector generation.

Labs:

Subword Tokenizer & Normalized Dense Embeddings

Python 3.13 • NumPy • PyTorch

Terminal Output

Click Run Code to execute this algorithm in the browser sandbox.

Finished this lesson?

Mark it as complete to record your progress and unlock the next module.