Elements of Information Theory — Cover & Thomas

Thomas M. Cover and Joy A. Thomas, 2nd Edition (Wiley)

The standard reference on information theory: entropy, relative entropy (KL divergence), and mutual information, along with the asymptotic equipartition property, data compression, and channel capacity. Entropy and the source-coding bound are the groundwork under the compression chapters of Course 1’s image and audio phases — the entropy coding behind image compression (Section 17) and the psychoacoustic audio coding of Section 19 — while cross-entropy and KL divergence remain the training objectives of the on-device classifiers and denoisers in Course 2.

Chapters

Exercises added as I work through each chapter.