d49d9e4038
## What changes were proposed in this pull request? This is a followup to reduce AppVeyor test time. This PR proposes to reduce the number of shuffle partitions to reduce the tasks running R workers in few particular tests. The symptom is similar as described in `https://github.com/apache/spark/pull/19722`. There are many R processes newly launched on Windows without forking and it makes the differences of elapsed time between Linux and Windows. Here is the simple comparison for before/after of this change. I manually tested this by disabling `spark.sparkr.use.daemon`. Disabling it resembles the tests on Windows: **Before** <img width="672" alt="2017-11-25 12 22 13" src="https://user-images.githubusercontent.com/6477701/33217949-b5528dfa-d17d-11e7-8050-75675c39eb20.png"> **After** <img width="682" alt="2017-11-25 12 32 00" src="https://user-images.githubusercontent.com/6477701/33217958-c6518052-d17d-11e7-9f8e-1be21a784559.png"> So, this probably will reduce roughly more than 10 minutes. ## How was this patch tested? AppVeyor tests Author: hyukjinkwon <gurwls223@gmail.com> Closes #19816 from HyukjinKwon/SPARK-21693-followup. |
||
---|---|---|
.. | ||
jarTest.R | ||
packageInAJarTest.R | ||
test_binary_function.R | ||
test_binaryFile.R | ||
test_broadcast.R | ||
test_client.R | ||
test_context.R | ||
test_includePackage.R | ||
test_jvm_api.R | ||
test_mllib_classification.R | ||
test_mllib_clustering.R | ||
test_mllib_fpm.R | ||
test_mllib_recommendation.R | ||
test_mllib_regression.R | ||
test_mllib_stat.R | ||
test_mllib_tree.R | ||
test_parallelize_collect.R | ||
test_rdd.R | ||
test_Serde.R | ||
test_shuffle.R | ||
test_sparkR.R | ||
test_sparkSQL.R | ||
test_streaming.R | ||
test_take.R | ||
test_textFile.R | ||
test_utils.R | ||
test_Windows.R |