Question

This question is about NDCG. Suppose a 5-point scale is used in relevance judgements, and a ranking of answers to a query is: 4, 3, 1, 0, 3, 2, 1, 4, 0, 2. What is the value of NDCG @ 10?

Answer 1

Step 1: Calculate the cumulative gain which is equal to sum of relevance score i.e ranking of answer

CG= 4+3+1+0+3+2+1+4+0+2= 20

Step 2: Calculate discounted cumulative gain.

DCG= where i = 1 to 10 and rel is relevance score

DCG= 4/log(2) + 3/log(3) + 1/log(4) + 0/log(5) + 3/log(6) + 2/log(7) + 1/log(8) + 4/log(9) + 0/log(10) + 2/log(11)

DCG= 4 + 1.89+ .5 + 0 + 1.16 + .712 + .33 + 1.26 + 0 + .578 = 10.43

Step 3: Arrange in descending order of their relevance score and Calculate Ideal discounted cumulative gain.

IDCG= 4/log(2) + 4/log(3) + 3/log(4) + 3/log(5) + 2/log(6) + 2/log(7) + 1/log(8) + 1/log(9) + 0/log(10) + 0/log(11)

IDCG= 4+ 2.52+ 1.5+ 1.29 + .77 + .712 + .33 + .315 + 0 + 0 = 11.437

Step 3: Calculate Normalised discounted cumulative gain(NDCG)

NDCG= DCG\IDCG = 10.43/11.437 = .912

NDCG@10 = .912 x 10 = 9.12

Thank You!