2

Generally there is a relevance associated with item in ndcg calculation but if we only have feedback in 0/1 form. Eg

list ={1,0,0,0,1} when we have recommended 5 items (first and last items are relevant here) How do we calculate ndcg here ? and does order matters in ndcg evaluation ?

Also what metrics are useful for evaluation in a binary feedback based recommendation ?

vaibhav
  • 164
  • 8

1 Answers1

1

NDCG@N (Normalized Discounted Comulative Gain) is a ranking measurement for top N position in your ranked output, so yes, the order matters the most here. Look at the good description how to calculate and for better understanding on the example on the wikipedia article. Just just {0,1} as a relevance score.

Other useful measures for evaluating ranked list outputs from recommender systems are: hit ratio (HR@N), recall and precision (REC@N), Mean Recipricol Rank (MRR@N), AUC, Mean Percentile Rank (MRP) or for graded relevance (not your case) Expected Reciprocal Rank (ERR)

Bartłomiej Twardowski
  • 602
  • 4
  • 9
  • 1
    what would IDCG be in the case of recommender system? Imagine for top-5, will we calculate IDCG for [1,1,1,1,1] expecting user to click on all 5? – Hardik Gupta Oct 24 '20 at 06:48