Question

Compute the entropy of a bit string where the probability of a 0 is 45% and the probability of a 1 is 55%. Compare the result, using the formula sum[ –pi log pi]; Do the same for a 50/50 probability. Comment on your results.

Answer 1

We have and .

The entropy is given by

For the 50/50 probability, that is when , the entropy is given by

So if we compare the two cases .

Or in other words entropy of 50/50 probability is more than entropy of the given case p0=0.45 and p1=0.55.

Also we can see that 50/50 probability is the discrete uniform case where random variable takes only two vales with same probability and entropy is maximum for discrete unifrom.