ArticlesProjectsWeeklyCredentialsAbout

TMC #0012: Information Theory: Shannon Entropy & Channel Capacity

A pure-Python implementation of Claude Shannon's 1948 information theory: Shannon entropy, mutual information, Shannon-Hartley channel capacity, BSC capacity, and optimal Huffman source coding.

shannoninformation-theoryentropychannel-capacityhuffmancoding-theorypython

Shannon's 1948 paper gave engineers the first rigorous answers to three questions: what is information, how efficiently can a source be encoded, and what is the maximum rate at which a noisy channel can carry information? This toolkit implements all three results.

What It Includes

Entropy functions: entropy(probs), joint_entropy(joint_probs), conditional_entropy(joint_probs), mutual_information(joint_probs). Demonstrates the entropy of a fair coin (1.0 bit), fair die (2.585 bits), and English letter distribution (~4.11 bits/letter vs a theoretical max of 4.70 bits).

Channel capacity: shannon_hartley_capacity(bandwidth_hz, snr_linear) implementing C=Blog⁡2(1+S/N)C = B \log_2(1 + S/N). Sweeps SNR from 0 dB to 40 dB for a 1 MHz channel; compares against practical modem speeds by decade. bsc_capacity(p) computes the Binary Symmetric Channel capacity C=1−Hb(p)C = 1 - H_b(p).

Huffman coder: huffman_codebook(frequencies) and average_code_length(codebook, frequencies). Builds an optimal prefix-free code for English letter frequencies and measures how close the average code length is to the entropy bound. Demonstrates that 'E' gets a 3-bit code and 'Z' gets an 8-bit code.

Running It

python information_theory.py                    # all demos
python information_theory.py --demo entropy     # entropy examples only
python information_theory.py --demo capacity    # channel capacity sweep
python information_theory.py --demo huffman     # Huffman coding demo

No external dependencies: pure Python 3.10+.

Source code
# TMC #0012 — Information Theory: Shannon Entropy & Channel Capacity

Companion code for [The Thinking Machine Chronicles #0012](/articles/tmc-0012-shannon-information-theory).

Implements Claude Shannon's three core results from "A Mathematical Theory of Communication" (1948):
1. Shannon entropy, joint entropy, conditional entropy, mutual information
2. Huffman source coding — demonstrates the source coding theorem bound
3. Shannon-Hartley channel capacity and Binary Symmetric Channel capacity

## Usage

```bash
python information_theory.py                    # all demos
python information_theory.py --demo entropy     # entropy examples
python information_theory.py --demo capacity    # channel capacity sweep
python information_theory.py --demo huffman     # Huffman coding
```

Pure Python 3.10+, no external dependencies.
← All projects