Josh Rosen 7956dd7ab0 [SPARK-7698] Cache and reuse buffers in ExecutorMemoryAllocator when using heap allocation ... When on-heap memory allocation is used, ExecutorMemoryManager should maintain a cache / pool of buffers for re-use by tasks. This will significantly improve the performance of the new Tungsten's sort-shuffle for jobs with many short-lived tasks by eliminating a major source of GC. This pull request is a minimum-viable-implementation of this idea. In its current form, this patch significantly improves performance on a stress test which launches huge numbers of short-lived shuffle map tasks back-to-back in the same JVM. Author: Josh Rosen <joshrosen@databricks.com> Closes #6227 from JoshRosen/SPARK-7698 and squashes the following commits: fd6cb55 [Josh Rosen] SoftReference -> WeakReference b154e86 [Josh Rosen] WIP sketch of pooling in ExecutorMemoryManager		2015-05-20 16:37:11 -07:00
..
src	[SPARK-7698] Cache and reuse buffers in ExecutorMemoryAllocator when using heap allocation	2015-05-20 16:37:11 -07:00
pom.xml	[SPARK-7081] Faster sort-based shuffle path using binary processing cache-aware sort	2015-05-13 17:07:31 -07:00