ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Added 2D canonical edge partitioner

Browse source
This commit is contained in:
Dan Crankshaw 2013-10-05 15:20:15 -07:00
parent da3e123afb
commit e096cbe90e
1 changed files with 15 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
graph/src/main/scala/org/apache/spark/graph/impl/GraphImpl.scala
View file

@ -361,7 +361,7 @@ object GraphImpl {
* Notice that P0 has many edges and as a consequence this * Notice that P0 has many edges and as a consequence this
* partitioning would lead to poor work balance. To improve * partitioning would lead to poor work balance. To improve
* balance we first multiply each vertex id by a large prime * balance we first multiply each vertex id by a large prime
* to effectively suffle the vertex locations. * to effectively shuffle the vertex locations.
* *
* One of the limitations of this approach is that the number of * One of the limitations of this approach is that the number of
* machines must either be a perfect square. We partially address * machines must either be a perfect square. We partially address
@ -381,6 +381,20 @@ object GraphImpl {
} }
protected def canonicalEdgePartitionFunction2D(srcOrig: Vid, dstOrig: Vid,
numParts: Pid, ceilSqrtNumParts: Pid): Pid = {
val mixingPrime: Vid = 1125899906842597L
// Partitions by canonical edge direction
// @todo(crankshaw) evaluate the cases
val src = math.min(srcOrig, dstOrig)
val dst = math.max(srcOrig, dstOrig)
val col: Pid = ((math.abs(src) * mixingPrime) % ceilSqrtNumParts).toInt
val row: Pid = ((math.abs(dst) * mixingPrime) % ceilSqrtNumParts).toInt
(col * ceilSqrtNumParts + row) % numParts
}
/** /**
* Create the edge table RDD, which is much more efficient for Java heap storage than the * Create the edge table RDD, which is much more efficient for Java heap storage than the
* normal edges data structure (RDD[(Vid, Vid, ED)]). * normal edges data structure (RDD[(Vid, Vid, ED)]).