ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-7490] [CORE] [Minor] MapOutputTracker.deserializeMapStatuses: close input streams

Browse source 
GZIPInputStream allocates native memory that is not freed until close() or
when the finalizer runs. It is best to close() these streams explicitly.

stephenh made the same change for serializeMapStatuses in commit b0d884f0. This is the same change for deserialize.

(I ran the unit test suite! it seems to have passed. I did not make a JIRA since this seems "trivial", and the guidelines suggest it is not required for trivial changes)

Author: Evan Jones <ejones@twitter.com>

Closes #5982 from evanj/master and squashes the following commits:

0d76e85 [Evan Jones] [CORE] MapOutputTracker.deserializeMapStatuses: close input streams

(cherry picked from commit 25889d8d97)
Signed-off-by: Sean Owen <sowen@cloudera.com>
This commit is contained in:
Evan Jones 2015-05-08 22:00:39 +01:00 committed by Sean Owen
parent 82be68f105
commit 62308097b2
1 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
core/src/main/scala/org/apache/spark/MapOutputTracker.scala
View file

@ -367,7 +367,11 @@ private[spark] object MapOutputTracker extends Logging {
// Opposite of serializeMapStatuses.
def deserializeMapStatuses(bytes: Array[Byte]): Array[MapStatus] = {
val objIn = new ObjectInputStream(new GZIPInputStream(new ByteArrayInputStream(bytes)))
objIn.readObject().asInstanceOf[Array[MapStatus]]
Utils.tryWithSafeFinally {
objIn.readObject().asInstanceOf[Array[MapStatus]]
} {
objIn.close()
}
}
// Convert an array of MapStatuses to locations and sizes for a given reduce ID. If