Eyal Farago 2028e5a82b [SPARK-21907][CORE] oom during spill ... ## What changes were proposed in this pull request? 1. a test reproducing [SPARK-21907](https://issues.apache.org/jira/browse/SPARK-21907) 2. a fix for the root cause of the issue. `org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill` calls `org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.reset` which may trigger another spill, when this happens the `array` member is already de-allocated but still referenced by the code, this causes the nested spill to fail with an NPE in `org.apache.spark.memory.TaskMemoryManager.getPage`. This patch introduces a reproduction in a test case and a fix, the fix simply sets the in-mem sorter's array member to an empty array before actually performing the allocation. This prevents the spilling code from 'touching' the de-allocated array. ## How was this patch tested? introduced a new test case: `org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterSuite#testOOMDuringSpill`. Author: Eyal Farago <eyal@nrgene.com> Closes #19181 from eyalfa/SPARK-21907__oom_during_spill.		2017-10-10 22:49:47 +02:00
..
org/apache/spark	[SPARK-21907][CORE] oom during spill	2017-10-10 22:49:47 +02:00
test/org/apache/spark	[SPARK-21578][CORE] Add JavaSparkContextSuite	2017-08-01 21:59:02 -07:00