Grid-based approach for distributed frequent itemsets mining using dynamic workload management
Résumé
Discovering frequent itemsets is a crucial task in data mining. This study presents a grid-based frequent itemsets generation approach taking into account the underlying platform nature, and exploiting an inherent property of the basic task, using the Apriori algorithm, related to the candidate sets generation. Gridbased implementations introduce constraints related to the communication and synchronization overheads, the platform heterogeneity, and the underlying middleware and tools. Other constraints are related to the datasets properties and distribution. In the proposed approach only a local pruning strategy is considered. This greatly reduces the communication and synchronization costs. A block-based approach is introduced for memory constraints and dynamic workload management. This paper describes this approach and evaluates its performance on large scale datasets on a widely distributed grid testbed. Our performance study shows that our approach greatly enhances the performance and achieves high scalability compared to the grid implementation of a distributed Apriori founded algorithm, namely the FDM approach.