High Performance Distributed Co-clustering and Collaborative Filtering

Petascale Analytics is a hot research area both in academia and industry. It envisages processing massive amounts of data at extremely high rates and generating new scientific insights along with positive impact (for both users and providers) of industries such as E-commerce, Telecom, Finance, Life Sciences and so forth. We consider Collaborative filtering (CF) and Clustering algorithms that are key fundamental analytic kernels that help in achieving these aims. Real-time CF and co-clustering on highly sparse massive datasets, while achieving a high prediction accuracy, is a computationally challenging problem. In this report, we present the novel designs for soft real-time (less than 1 min.) distributed co-clustering based Collaborative Filtering algorithm. Our distributed algorithms have been optimized for multi-core cluster architectures. Theoretical analysis of time complexity of our algorithms proves the efficacy of our approach. Using Netflix (900M training ratings with replication) as well as the Yahoo KDD Cup 1 (4.6B training ratings with replication) datasets , we demonstrate the performance and scalability of our flat and hierarchical algorithm on a 1024 & 4096-node multi-core cluster architecture respectively. Our hybrid distributed flat algorithm performs 2x better than the previous flat algorithms on 1024 nodes of BlueGene/P while our batch mode hierarchical distributed algorithm (implemented using OpenMP with MPI) demonstrates around 4x better performance (on Blue Gene/P) as compared to the our hybrid distributed flat algorithm [best prior (flat MPI+OMP based algorithm) work, along with high accuracy (26± 4 RMSE for Yahoo KDD Cup data and 87± 0.02 for Netflix data). For online execution, we demonstrate around 3x gain vs the online baseline MPI algorithm. To the best of our knowledge, these are the best known performance results for collaborative filtering, at high prediction accuracy, for multi-core cluster architectures.