[SPARK-9641] [DOCS] spark.shuffle.service.port is not documented
Document spark.shuffle.service.{enabled,port} CC sryza tgravescs This is pretty minimal; is there more to say here about the service? Author: Sean Owen <sowen@cloudera.com> Closes #7991 from srowen/SPARK-9641 and squashes the following commits: 3bb946e [Sean Owen] Add link to docs for setup and config of external shuffle service 2302e01 [Sean Owen] Document spark.shuffle.service.{enabled,port}
This commit is contained in:
parent
cdd53b762b
commit
0d7aac99da
|
@ -473,6 +473,25 @@ Apart from these, the following properties are also available, and may be useful
|
||||||
<code>spark.storage.memoryFraction</code>.
|
<code>spark.storage.memoryFraction</code>.
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>spark.shuffle.service.enabled</code></td>
|
||||||
|
<td>false</td>
|
||||||
|
<td>
|
||||||
|
Enables the external shuffle service. This service preserves the shuffle files written by
|
||||||
|
executors so the executors can be safely removed. This must be enabled if
|
||||||
|
<code>spark.dynamicAllocation.enabled</code> is "true". The external shuffle service
|
||||||
|
must be set up in order to enable it. See
|
||||||
|
<a href="job-scheduling.html#configuration-and-setup">dynamic allocation
|
||||||
|
configuration and setup documentation</a> for more information.
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>spark.shuffle.service.port</code></td>
|
||||||
|
<td>7337</td>
|
||||||
|
<td>
|
||||||
|
Port on which the external shuffle service will run.
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td><code>spark.shuffle.sort.bypassMergeThreshold</code></td>
|
<td><code>spark.shuffle.sort.bypassMergeThreshold</code></td>
|
||||||
<td>200</td>
|
<td>200</td>
|
||||||
|
|
Loading…
Reference in a new issue