ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Updating docs to include missing information about reducers and clarify ...

Browse source 
...how the OFFHEAP storage level works (there has been confusion around this).

Author: Ali Ghodsi <alig@cs.berkeley.edu>

Closes #1089 from alig/master and squashes the following commits:

ca8114d [Ali Ghodsi] Updating docs to include missing information about reducers and clarify how the OFFHEAP storage level works (there has been confusion around this).
This commit is contained in:
Ali Ghodsi 2014-06-15 23:44:30 -07:00 committed by Patrick Wendell
parent 9672ee07fb
commit 119b06a04f
1 changed files with 5 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
docs/programming-guide.md
View file

@ -899,7 +899,7 @@ for details.
</tr>
<tr>
<td> <b>reduceByKey</b>(<i>func</i>, [<i>numTasks</i>]) </td>
<td> When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function. Like in <code>groupByKey</code>, the number of reduce tasks is configurable through an optional second argument. </td>
<td> When called on a dataset of (K, V) pairs, returns a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function <i>func</i>, which must be of type (V,V) => V. Like in <code>groupByKey</code>, the number of reduce tasks is configurable through an optional second argument. </td>
</tr>
<tr>
<td> <b>aggregateByKey</b>(<i>zeroValue</i>)(<i>seqOp</i>, <i>combOp</i>, [<i>numTasks</i>]) </td>
@ -1067,7 +1067,10 @@ storage levels is:
<td> Store RDD in serialized format in <a href="http://tachyon-project.org">Tachyon</a>.
Compared to MEMORY_ONLY_SER, OFF_HEAP reduces garbage collection overhead and allows executors
to be smaller and to share a pool of memory, making it attractive in environments with
large heaps or multiple concurrent applications.
large heaps or multiple concurrent applications. Furthermore, as the RDDs reside in Tachyon,
the crash of an executor does not lead to losing the in-memory cache. In this mode, the memory
in Tachyon is discardable. Thus, Tachyon does not attempt to reconstruct a block that it evicts
from memory.
</td>
</tr>
</table>