Scaling Out Link Prediction with SNAPLE: 1 Billion Edges and Beyond
Abstract
In this paper, we consider how the emblematic problem of link-prediction can be implemented
efficiently in gather-apply-scatter (GAS) platforms, a popular distributed graph-computation model. Our
proposal, called S NAPLE , exploits a novel highly-localized vertex scoring technique, and minimizes the
cost of data flow while maintaining prediction quality.
When used within GraphLab, S NAPLE can scale to extremely large graphs that a standard implementation
of link prediction on GraphLab cannot handle. More precisely, we show that S NAPLE can process a graph
containing 1.4 billions edges on a 256 cores cluster in less than three minutes, with no penalty in the quality
of predictions. This result corresponds to an over-linear speedup of 30 against a 20-core standalone machine
running a non-distributed state-of-the-art solution.
Origin | Files produced by the author(s) |
---|
Loading...