fa5dc0a45a
## What changes were proposed in this pull request? For the below three thread configuration items applied to both driver and executor, spark.rpc.io.serverThreads spark.rpc.io.clientThreads spark.rpc.netty.dispatcher.numThreads, we separate them to driver specifics and executor specifics. spark.driver.rpc.io.serverThreads < - > spark.executor.rpc.io.serverThreads spark.driver.rpc.io.clientThreads < - > spark.executor.rpc.io.clientThreads spark.driver.rpc.netty.dispatcher.numThreads < - > spark.executor.rpc.netty.dispatcher.numThreads Spark reads these specifics first and fall back to the common configurations. ## How was this patch tested? We ran the SimpleMap app without shuffle for benchmark purpose to test Spark's scalability in HPC with omini-path NIC which has higher bandwidth than normal ethernet NIC. Spark's base version is 2.4.0. Spark ran in the Standalone mode. Driver was in a standalone node. After the separation, the performance is improved a lot in 256 nodes and 512 nodes. see below test results of SimpleMapTask before and after the enhancement. You can view the tables in the [JIRA](https://issues.apache.org/jira/browse/SPARK-26632) too. ds: spark.driver.rpc.io.serverThreads dc: spark.driver.rpc.io.clientThreads dd: spark.driver.rpc.netty.dispatcher.numThreads ed: spark.executor.rpc.netty.dispatcher.numThreads time: Overall Time (s) old time: Overall Time without Separation (s) **Before:** nodes | ds | dc | dd | ed | time -- |-- | -- | -- | -- | -- 128 nodes | 8 | 8 | 8 | 8 | 108 256 nodes | 8 | 8 | 8 | 8 | 196 512 nodes | 8 | 8 | 8 | 8 | 377 **After:** nodes | ds | dc | dd | ed | time | improvement -- | -- | -- | -- | -- | -- | -- 128 nodes | 15 | 15 | 10 | 30 | 107 | 0.9% 256 nodes | 12 | 15 | 10 | 30 | 159 | 18.8% 512 nodes | 12 | 15 | 10 | 30 | 283 | 24.9% Closes #23560 from zjf2012/thread_conf_separation. Authored-by: jiafu.zhang@intel.com <jiafu.zhang@intel.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> |
||
|---|---|---|
| .. | ||
| build.properties | ||
| MimaBuild.scala | ||
| MimaExcludes.scala | ||
| plugins.sbt | ||
| SparkBuild.scala | ||