ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-22454][CORE] ExternalShuffleClient.close() should check clientFactory null

Browse source 
## What changes were proposed in this pull request?

`ExternalShuffleClient.close()` should check `clientFactory` null. otherwise it will throw NPE sometimes:
```
17/11/06 20:08:05 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
	at org.apache.spark.network.shuffle.ExternalShuffleClient.close(ExternalShuffleClient.java:152)
	at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1407)
	at org.apache.spark.SparkEnv.stop(SparkEnv.scala:89)
	at org.apache.spark.SparkContext$$anonfun$stop$11.apply$mcV$sp(SparkContext.scala:1849)
```

## How was this patch tested?
manual tests

Author: Yuming Wang <wgyumg@gmail.com>

Closes #19670 from wangyum/SPARK-22454.
This commit is contained in:
Yuming Wang 2017-11-07 08:30:58 +00:00 committed by Sean Owen
parent 14a32a647a
commit 9df08e218c
1 changed files with 4 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleClient.java
View file

@ -148,6 +148,9 @@ public class ExternalShuffleClient extends ShuffleClient {
@Override
public void close() {
checkInit();
clientFactory.close();
if (clientFactory != null) {
clientFactory.close();
clientFactory = null;
}
}
}