Week 28 — Coding Projects

Core

Compute entropy, cross-entropy, and KL divergence over large batches.

• NumPy: Implement entropy for discrete distributions. Implement KL(p ‖ q) with care around zeros. Compare and plot.
• Metal: Per-row entropy/KL compute kernel as a batched reduction exercise. · Reading: MBT — reductions and per-buffer statistical kernels.
• Vulkan: Batched reduction-like numerical kernel for entropy/KL. · Reading: Vulkan Book — batched reduction-like numerical kernels.
• CUDA: Efficient row-wise reductions with log-heavy compute. · Reading: CUDA Book — efficient row-wise reductions and log-heavy compute.
• Stretch: Add softmax + cross-entropy pipeline. Interpret classification-style loss examples.
• Verify: Entropy is nonnegative · KL is near zero only when distributions match · Numerical stability around tiny probabilities is handled carefully.