ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[MINOR][CORE] Add test cases for RDD.cartesian

Browse source 
## What changes were proposed in this pull request?

While looking through the codebase, it appeared that the scala code for RDD.cartesian does not have any tests for correctness. This adds a couple basic tests to verify cartesian yields correct values. While the implementation for RDD.cartesian is pretty simple, it always helps to have a few tests!

## How was this patch tested?

The new test cases pass, and the scala style tests from running dev/run-tests all pass.

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: Nihar Sheth <niharrsheth@gmail.com>

Closes #21765 from NiharS/cartesianTests.
This commit is contained in:
Nihar Sheth 2018-07-18 09:14:36 -05:00 committed by Sean Owen
parent 34cb3b54e9
commit 2694dd2bf0
1 changed files with 23 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
View file

@ -443,7 +443,7 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
map{x => List(x)}.toList, "Tried coalescing 9 partitions to 20 but didn't get 9 back")
}
test("coalesced RDDs with partial locality") {
test("coalesced RDDs with partial locality") {
// Make an RDD that has some locality preferences and some without. This can happen
// with UnionRDD
val data = sc.makeRDD((1 to 9).map(i => {
@ -846,6 +846,28 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
assert(partitions(1) === Seq((1, 3), (3, 8), (3, 8)))
}
test("cartesian on empty RDD") {
val a = sc.emptyRDD[Int]
val b = sc.parallelize(1 to 3)
val cartesian_result = Array.empty[(Int, Int)]
assert(a.cartesian(a).collect().toList === cartesian_result)
assert(a.cartesian(b).collect().toList === cartesian_result)
assert(b.cartesian(a).collect().toList === cartesian_result)
}
test("cartesian on non-empty RDDs") {
val a = sc.parallelize(1 to 3)
val b = sc.parallelize(2 to 4)
val c = sc.parallelize(1 to 1)
val a_cartesian_b =
Array((1, 2), (1, 3), (1, 4), (2, 2), (2, 3), (2, 4), (3, 2), (3, 3), (3, 4))
val a_cartesian_c = Array((1, 1), (2, 1), (3, 1))
val c_cartesian_a = Array((1, 1), (1, 2), (1, 3))
assert(a.cartesian[Int](b).collect().toList.sorted === a_cartesian_b)
assert(a.cartesian[Int](c).collect().toList.sorted === a_cartesian_c)
assert(c.cartesian[Int](a).collect().toList.sorted === c_cartesian_a)
}
test("intersection") {
val all = sc.parallelize(1 to 10)
val evens = sc.parallelize(2 to 10 by 2)