spark-instrumented-optimizer/core
Akshat Aranya a16357413d [SPARK-7795] [CORE] Speed up task scheduling in standalone mode by reusing serializer
My experiments with scheduling very short tasks in standalone cluster mode indicated that a significant amount of time was being spent in scheduling the tasks (>500ms for 256 tasks).  I found that most of the time was being spent in creating a new instance of serializer for each task.  Changing this to just one serializer brought down the scheduling time to 8ms.

Author: Akshat Aranya <aaranya@quantcast.com>

Closes #6323 from coolfrood/master and squashes the following commits:

12d8c9e [Akshat Aranya] Reduce visibility of serializer
bd4a5dd [Akshat Aranya] Style fix
0b8ca93 [Akshat Aranya] Incorporate review comments
fe530cd [Akshat Aranya] Speed up task scheduling in standalone mode by reusing serializer instead of creating a new one for each task.
2015-05-22 22:03:31 -07:00
..
src [SPARK-7795] [CORE] Speed up task scheduling in standalone mode by reusing serializer 2015-05-22 22:03:31 -07:00
pom.xml [SPARK-7081] Faster sort-based shuffle path using binary processing cache-aware sort 2015-05-13 17:07:31 -07:00