Hieu Huynh 51e2b38d93 [SPARK-24992][CORE] spark should randomize yarn local dir selection ... **Description: [SPARK-24992](https://issues.apache.org/jira/browse/SPARK-24992)** Utils.getLocalDir is used to get path of a temporary directory. However, it always returns the the same directory, which is the first element in the array localRootDirs. When running on YARN, this might causes the case that we always write to one disk, which makes it busy while other disks are free. We should randomize the selection to spread out the loads. **What changes were proposed in this pull request?** This PR randomized the selection of local directory inside the method Utils.getLocalDir. This change affects the Utils.fetchFile method since it based on the fact that Utils.getLocalDir always return the same directory to cache file. Therefore, a new variable cachedLocalDir is used to cache the first localDirectory that it gets from Utils.getLocalDir. Also, when getting the configured local directories (inside Utils. getConfiguredLocalDirs), in case we are in yarn mode, the array of directories are also randomized before return. Author: Hieu Huynh <“Hieu.huynh@oath.com”> Closes #21953 from hthuynh2/SPARK_24992.		2018-08-06 13:58:28 -05:00
..
src	[SPARK-24992][CORE] spark should randomize yarn local dir selection	2018-08-06 13:58:28 -05:00
pom.xml	[SPARK-24420][BUILD] Upgrade ASM to 6.1 to support JDK9+	2018-07-03 10:13:48 -07:00