Large Language Models & Generative AITokenization, Context Windows & High-Dimensional Embeddings
Written Guide20 minutes

Tokenization, Context Windows & High-Dimensional Embeddings

Lesson Summary: Byte-Pair Encoding (BPE), SentencePiece, vector representations, and semantic distance metrics.

Vector Embeddings & Semantic Cosine Distance Lab

LLMs & RAGLesson / Article Lab

Compute dot product similarity scores, normalize high-dimensional vectors, and rank semantic text chunks.

Subword Tokenizer & Normalized Dense Embeddings
Pyodide Wasm
Interactive Challenge: Modify code inputs, click Run Code to execute live in WebAssembly.
+25 XP Reward
Wasm Terminal Output

Click Run Code to execute this algorithm in the browser sandbox.

Language models do not read text directly—they process discrete numerical identifiers called tokens. In this lesson, we trace how strings convert into token embeddings in high-dimensional vector spaces (e.g. 1536 dimensions for OpenAI text-embedding-3-small).

Cosine Similarity

To measure semantic similarity between two normalized embeddings u\mathbf{u} and v\mathbf{v}:

Cosine Similarity=u⋅v∥u∥∥v∥\text{Cosine Similarity} = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}

Next Lesson
System Prompts, Few-Shot In-Context Learning & Structured Outputs