Updating docs to include missing information about reducers and clarify ...
...how the OFFHEAP storage level works (there has been confusion around this). Author: Ali Ghodsi <alig@cs.berkeley.edu> Closes #1089 from alig/master and squashes the following commits: ca8114d [Ali Ghodsi] Updating docs to include missing information about reducers and clarify how the OFFHEAP storage level works (there has been confusion around this).
This commit is contained in:
parent
9672ee07fb
commit
119b06a04f
|
@ -899,7 +899,7 @@ for details.
|
|||
</tr>
|
||||
<tr>
|
||||
<td> <b>reduceByKey</b>(<i>func</i>, [<i>numTasks</i>]) </td>
|
||||
<td> When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function. Like in <code>groupByKey</code>, the number of reduce tasks is configurable through an optional second argument. </td>
|
||||
<td> When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function <i>func</i>, which must be of type (V,V) => V. Like in <code>groupByKey</code>, the number of reduce tasks is configurable through an optional second argument. </td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td> <b>aggregateByKey</b>(<i>zeroValue</i>)(<i>seqOp</i>, <i>combOp</i>, [<i>numTasks</i>]) </td>
|
||||
|
@ -1067,7 +1067,10 @@ storage levels is:
|
|||
<td> Store RDD in serialized format in <a href="http://tachyon-project.org">Tachyon</a>.
|
||||
Compared to MEMORY_ONLY_SER, OFF_HEAP reduces garbage collection overhead and allows executors
|
||||
to be smaller and to share a pool of memory, making it attractive in environments with
|
||||
large heaps or multiple concurrent applications.
|
||||
large heaps or multiple concurrent applications. Furthermore, as the RDDs reside in Tachyon,
|
||||
the crash of an executor does not lead to losing the in-memory cache. In this mode, the memory
|
||||
in Tachyon is discardable. Thus, Tachyon does not attempt to reconstruct a block that it evicts
|
||||
from memory.
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
|
Loading…
Reference in a new issue