Josh Rosen 7cea552e1e [SPARK-7698] Cache and reuse buffers in ExecutorMemoryAllocator when using heap allocation ... When on-heap memory allocation is used, ExecutorMemoryManager should maintain a cache / pool of buffers for re-use by tasks. This will significantly improve the performance of the new Tungsten's sort-shuffle for jobs with many short-lived tasks by eliminating a major source of GC. This pull request is a minimum-viable-implementation of this idea. In its current form, this patch significantly improves performance on a stress test which launches huge numbers of short-lived shuffle map tasks back-to-back in the same JVM. Author: Josh Rosen <joshrosen@databricks.com> Closes #6227 from JoshRosen/SPARK-7698 and squashes the following commits: fd6cb55 [Josh Rosen] SoftReference -> WeakReference b154e86 [Josh Rosen] WIP sketch of pooling in ExecutorMemoryManager (cherry picked from commit 7956dd7ab0) Signed-off-by: Josh Rosen <joshrosen@databricks.com>		2015-05-20 16:39:36 -07:00
..
src	[SPARK-7698] Cache and reuse buffers in ExecutorMemoryAllocator when using heap allocation	2015-05-20 16:39:36 -07:00
pom.xml	Preparing development version 1.4.0-SNAPSHOT	2015-05-20 16:30:01 -07:00