Matei Zaharia 4fde28c206 SPARK-2711. Create a ShuffleMemoryManager to track memory for all spilling collections ... This tracks memory properly if there are multiple spilling collections in the same task (which was a problem before), and also implements an algorithm that lets each thread grow up to 1 / 2N of the memory pool (where N is the number of threads) before spilling, which avoids an inefficiency with small spills we had before (some threads would spill many times at 0-1 MB because the pool was allocated elsewhere). Author: Matei Zaharia <matei@databricks.com> Closes #1707 from mateiz/spark-2711 and squashes the following commits: debf75b [Matei Zaharia] Review comments 24f28f3 [Matei Zaharia] Small rename c8f3a8b [Matei Zaharia] Update ShuffleMemoryManager to be able to partially grant requests 315e3a5 [Matei Zaharia] Some review comments b810120 [Matei Zaharia] Create central manager to track memory for all spilling collections		2014-08-04 23:41:03 -07:00
..
java/org/apache/spark	SPARK-2028: Expose mapPartitionsWithInputSplit in HadoopRDD	2014-07-31 11:35:38 -07:00
resources	SPARK-1798. Tests should clean up temp files	2014-05-12 14:16:19 -07:00
scala/org/apache/spark	SPARK-2711. Create a ShuffleMemoryManager to track memory for all spilling collections	2014-08-04 23:41:03 -07:00