ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-16138] Try to cancel executor requests only if we have at least 1

Browse source 
## What changes were proposed in this pull request?
Adding additional check to if statement

## How was this patch tested?
I built and deployed to internal cluster to observe behaviour. After the change the invalid logging is gone:

```
16/06/22 08:46:36 INFO yarn.YarnAllocator: Driver requested a total number of 1 executor(s).
16/06/22 08:46:36 INFO yarn.YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 1 executors.
16/06/22 08:46:36 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
16/06/22 08:47:36 INFO yarn.ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 1.
```

Author: Peter Ableda <abledapeter@gmail.com>

Closes #13850 from peterableda/patch-2.
This commit is contained in:
Peter Ableda 2016-06-23 09:00:31 -05:00 committed by Tom Graves
parent 5eef1e6c6a
commit 5bf2889bfc
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
View file

@ -352,7 +352,7 @@ private[yarn] class YarnAllocator(
logInfo(s"Submitted container request (host: ${hostStr(request)}, capability: $resource)")
}
} else if (missing < 0) {
} else if (numPendingAllocate > 0 && missing < 0) {
val numToCancel = math.min(numPendingAllocate, -missing)
logInfo(s"Canceling requests for $numToCancel executor container(s) to have a new desired " +
s"total $targetNumExecutors executors.")