ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-27333][TEST] Update thread audit whitelist to skip broadcast-exchange-.*, process reaper and StatisticsDataReferenceCleaner threads

Browse source 
## What changes were proposed in this pull request?

Update thread audit whitelist to skip threads of the global broadcast exchange thread pool, process reaper and Hadoop FS statistics data reference cleaner thread.

## How was this patch tested?

Via existing UT using broadcast exchange via `sbt` i.e:

```
> project sql
> testOnly *.SessionStateSuite -- -z "fork new sessions and run query on inherited table"
```

Before (wrapped long line for manually to save horizontal scrolling for reviewers):

```
===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.SessionStateSuite,
thread names: broadcast-exchange-6, broadcast-exchange-0,
broadcast-exchange-2, broadcast-exchange-5, broadcast-exchange-7,
broadcast-exchange-4, broadcast-exchange-1, process reaper, broadcast-exchange-3,
 org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner =====
```

After this change no possible thread leak detected.

Closes #24244 from attilapiros/thread-audit-minor.

Authored-by: “attilapiros” <piros.attila.zsolt@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
“attilapiros” 2019-03-31 17:33:31 -07:00 committed by Dongjoon Hyun
parent fc9aad0957
commit 9eb896cc3b
1 changed files with 19 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
core/src/test/scala/org/apache/spark/ThreadAudit.scala
View file

@ -77,7 +77,25 @@ trait ThreadAudit extends Logging {
* asynchronously. In each case proper stopping is checked manually.
*/
"shuffle-client.*",
"shuffle-server.*"
"shuffle-server.*",
/**
* Global cleaner thread that manage statistics data references of Hadoop filesystems.
* This is excluded because their lifecycle is handled by Hadoop and spark has no explicit
* effect on it.
*/
"org.apache.hadoop.fs.FileSystem\\$Statistics\\$StatisticsDataReferenceCleaner",
/**
* A global thread pool for broadcast exchange executions.
*/
"broadcast-exchange.*",
/**
* A thread started by JRE to support safe parallel execution of waitFor() and exitStatus()
* methods to forked subprocesses.
*/
"process reaper"
)
private var threadNamesSnapshot: Set[String] = Set.empty