WSGP: A Window-based Streaming Graph Partitioning Approach - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

WSGP: A Window-based Streaming Graph Partitioning Approach

Chuanyou Li
  • Function : Author
  • PersonId : 1097898
Philippe Raipin Parvédy
  • Function : Author
  • PersonId : 960999

Abstract

Graph partitioning, a preliminary step of distributed graph processing, has been attracting increasing attention in the last decade. A high quality graph partitioning algorithm should facilitate graph processing by minimizing the communication overhead and maintaining the load balancing among distributed computing units. Offline partitioning algorithms usually require the knowledge of a complete graph,and therefore, are not adaptive to handle massive graph-structured data. On the contrary, streaming partitioning algorithms take edges or vertices as a stream and make partitioning decisions on the fly. However, the streaming manner faces dilemmas from time to time because of a lack of knowledge. Furthermore, an unmindful partitioning decision in such a dilemma could significantly decrease the partition quality. In this paper, we propose a novel window-based streaming graph partitioning algorithm (WSGP). WSGP leverages a greedy-based heuristic to perform edge partitioning. When facing a decision dilemma, WSGP utilizes a size-bounded window to buffer the edges. When the window is fully filled, an edge is poped and assigned to a partition. The assignment is decided by knowledge obtained from both the edges already settled and the ones still cached in the buffer window. Our experiments take into account various real-world benchmark graphs. The experimental results demonstrate that WSGP consistently has a smaller replication factor than the state-of-the-art algorithms by up to 23%, at a limited cost in terms of memory and comprehensive running time.
Fichier principal
Vignette du fichier
paper198_revise.pdf (983.46 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03217558 , version 1 (04-05-2021)

Identifiers

  • HAL Id : hal-03217558 , version 1

Cite

Yunbo Li, Chuanyou Li, Anne-Cécile Orgerie, Philippe Raipin Parvédy. WSGP: A Window-based Streaming Graph Partitioning Approach. CCGrid 2021 -21th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing, May 2021, Melbourne, Australia. pp.1-10. ⟨hal-03217558⟩
111 View
159 Download

Share

Gmail Mastodon Facebook X LinkedIn More