ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-5064][GraphX] Add numEdges upperbound validation for R-MAT graph generator to prevent infinite loop

Browse source 
I looked into GraphGenerators#chooseCell, and found that chooseCell can't generate more edges than pow(2, (2 * (log2(numVertices)-1))) to make a Power-law graph. (Ex. numVertices:4 upperbound:4, numVertices:8 upperbound:16, numVertices:16 upperbound:64)
If we request more edges over the upperbound, rmatGraph fall into infinite loop. So, how about adding an argument validation?

Author: Kenji Kikushima <kikushima.kenji@lab.ntt.co.jp>

Closes #3950 from kj-ki/SPARK-5064 and squashes the following commits:

4ee18c7 [Ankur Dave] Reword error message and add unit test
d760bc7 [Kenji Kikushima] Add numEdges upperbound validation for R-MAT graph generator to prevent infinite loop.
This commit is contained in:
Kenji Kikushima 2015-01-21 12:34:00 -08:00 committed by Ankur Dave
parent 7450a992b3
commit 3ee3ab592e
2 changed files with 16 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
graphx/src/main/scala/org/apache/spark/graphx/util/GraphGenerators.scala
View file

@ -133,6 +133,12 @@ object GraphGenerators {
// This ensures that the 4 quadrants are the same size at all recursion levels
val numVertices = math.round(
math.pow(2.0, math.ceil(math.log(requestedNumVertices) / math.log(2.0)))).toInt
val numEdgesUpperBound =
math.pow(2.0, 2 * ((math.log(numVertices) / math.log(2.0)) - 1)).toInt
if (numEdgesUpperBound < numEdges) {
throw new IllegalArgumentException(
s"numEdges must be <= $numEdgesUpperBound but was $numEdges")
}
var edges: Set[Edge[Int]] = Set()
while (edges.size < numEdges) {
if (edges.size % 100 == 0) {

10
graphx/src/test/scala/org/apache/spark/graphx/util/GraphGeneratorsSuite.scala
View file

@ -110,4 +110,14 @@ class GraphGeneratorsSuite extends FunSuite with LocalSparkContext {
}
}
test("SPARK-5064 GraphGenerators.rmatGraph numEdges upper bound") {
withSpark { sc =>
val g1 = GraphGenerators.rmatGraph(sc, 4, 4)
assert(g1.edges.count() === 4)
intercept[IllegalArgumentException] {
val g2 = GraphGenerators.rmatGraph(sc, 4, 8)
}
}
}
}