Shannon's H

Frank E. Ritter

22 Aug 2000


Explain H= - sum ( Pi Log(Pi) )

Where Pi is the probability of each event, and log is computed base 2.



1. Given 8 different characters (e.g. a-h), compute how many bits of information there are. (assume an equal distribution of characters).


2. Find the distribution of letters in the english language (from the net, from a book, by counting them even in this document). How much information is there in a single letter typically? How does this compare to the 7 and 8 bit ascii characters?


3. Assume that there are 9 postal zones in Nottingham England, NG1 to NG9. The mail to these zones is distributed as follows:

1 10%

2 10%

3 20%

4 30%

5 5%

6 5%

7 5%

8 5%

9 5%

(a) How much information is there in a typical postal code in Nottingham?

(b) How might you redesign the codes to make them better in some sense?


4. Find the Morse code. Examine its design using Shannon's H (if you can) and the distribution of the English language. Is it well designed? Could it be improved?

[Hint: compute the information of each character and compare it to its length. Perhaps draw a tree of how the codes are related.]