jerryshao 4482ff23ad [SPARK-17321][YARN] Avoid writing shuffle metadata to disk if NM recovery is disabled ... In the current code, if NM recovery is not enabled then `YarnShuffleService` will write shuffle metadata to NM local dir-1, if this local dir-1 is on bad disk, then `YarnShuffleService` will be failed to start. So to solve this issue, in Spark side if NM recovery is not enabled, then Spark will not persist data into leveldb, in that case yarn shuffle service can still be served but lose the ability for recovery, (it is fine because the failure of NM will kill the containers as well as applications). Tested in the local cluster with NM recovery off and on to see if folder is created or not. MiniCluster UT isn't added because in MiniCluster NM will always set port to 0, but NM recovery requires non-ephemeral port. Author: jerryshao <sshao@hortonworks.com> Closes #19032 from jerryshao/SPARK-17321. Change-Id: I8f2fe73d175e2ad2c4e380caede3873e0192d027		2017-08-31 09:26:20 +08:00
..
kvstore	[MINOR][BUILD] Fix build warnings and Java lint errors	2017-08-25 16:07:13 +01:00
network-common	[MINOR][BUILD] Fix build warnings and Java lint errors	2017-08-25 16:07:13 +01:00
network-shuffle	[SPARK-21501] Change CacheLoader to limit entries based on memory footprint	2017-08-23 11:51:11 -05:00
network-yarn	[SPARK-17321][YARN] Avoid writing shuffle metadata to disk if NM recovery is disabled	2017-08-31 09:26:20 +08:00
sketch	[SPARK-21592][BUILD] Skip maven-compiler-plugin main and test compilations in Maven build	2017-08-01 19:03:34 +01:00
tags	[SPARK-20453] Bump master branch version to 2.3.0-SNAPSHOT	2017-04-24 21:48:04 -07:00
unsafe	[MINOR][TEST] Off -heap memory leaks for unit tests	2017-08-30 10:16:11 +01:00