Comparative Recommender System Evaluation: Benchmarking Recommendation Frameworks

TitleComparative Recommender System Evaluation: Benchmarking Recommendation Frameworks
Publication TypeConference Paper
Year of Publication2014
AuthorsSaid, A, Bellogín, A
Refereed DesignationRefereed
Conference NameProceedings of the 8th ACM Conference on Recommender Systems
Conference LocationFoster City, CA, USA

Recommender systems research is often based on comparisons of predictive accuracy: the better the evaluation scores, the better the recommender. However, it is difficult to compare results from different recommender systems due to the many options in design and implementation of an evaluation strategy. Additionally, algorithm implementations can diverge from the standard formulation due to manual tuning and modifications that work better in some situations. In this work we compare common recommendation algorithms as implemented in three popular recommendation frameworks.% used by industry and academia. To provide a fair comparison, we have complete control of the evaluation dimensions being benchmarked: dataset, data splitting, evaluation strategies, and metrics. We also include results using the internal evaluation mechanisms of these frameworks. Our analysis points to large differences in recommendation accuracy across frameworks and strategies, i.e. the same baselines may perform orders of magnitude better or worse across frameworks. Our results show the necessity of clear guidelines when reporting evaluation of recommender systems to ensure reproducibility and comparison of results.