[SPARK-32920][FOLLOW-UP][CORE] Shutdown shuffleMergeFinalizeScheduler when DAGScheduler stop

### What changes were proposed in this pull request?

Call `shuffleMergeFinalizeScheduler.shutdownNow()` in `DAGScheduler.stop()`.

### Why are the changes needed?

Avoid the thread leak.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass existing tests.

Closes #33495 from Ngone51/SPARK-32920-followup.

Authored-by: yi.wu <yi.wu@databricks.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This commit is contained in:
yi.wu 2021-07-24 17:40:47 -07:00 committed by Dongjoon Hyun
parent 70a15868fc
commit 21450b3254

View file

@ -2512,6 +2512,7 @@ private[spark] class DAGScheduler(
def stop(): Unit = {
messageScheduler.shutdownNow()
shuffleMergeFinalizeScheduler.shutdownNow()
eventProcessLoop.stop()
taskScheduler.stop()
}