[SPARK-23431][CORE] Expose stage level peak executor metrics via REST API
### What changes were proposed in this pull request?
Note that this PR is forked from #23340 originally written by edwinalu.
This PR proposes to expose the peak executor metrics at the stage level via the REST APIs:
* `/applications/<application_id>/stages/`: peak values of executor metrics for **each stage**
* `/applications/<application_id>/stages/<stage_id>/< stage_attempt_id >`: peak values of executor metrics for **each executor** for the stage, followed by peak values of executor metrics for the stage
### Why are the changes needed?
The stage level peak executor metrics can help better understand your application's resource utilization.
### Does this PR introduce _any_ user-facing change?
1. For the `/applications/<application_id>/stages/` API, you will see the following new info for **each stage**:
```JSON
"peakExecutorMetrics" : {
"JVMHeapMemory" : 213367864,
"JVMOffHeapMemory" : 189011656,
"OnHeapExecutionMemory" : 0,
"OffHeapExecutionMemory" : 0,
"OnHeapStorageMemory" : 2133349,
"OffHeapStorageMemory" : 0,
"OnHeapUnifiedMemory" : 2133349,
"OffHeapUnifiedMemory" : 0,
"DirectPoolMemory" : 282024,
"MappedPoolMemory" : 0,
"ProcessTreeJVMVMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
"ProcessTreePythonVMemory" : 0,
"ProcessTreePythonRSSMemory" : 0,
"ProcessTreeOtherVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
"MinorGCCount" : 13,
"MinorGCTime" : 115,
"MajorGCCount" : 4,
"MajorGCTime" : 339
}
```
2. For the `/applications/<application_id>/stages/<stage_id>/<stage_attempt_id>` API, you will see the following new info for **each executor** under `executorSummary`:
```JSON
"peakMemoryMetrics" : {
"JVMHeapMemory" : 0,
"JVMOffHeapMemory" : 0,
"OnHeapExecutionMemory" : 0,
"OffHeapExecutionMemory" : 0,
"OnHeapStorageMemory" : 0,
"OffHeapStorageMemory" : 0,
"OnHeapUnifiedMemory" : 0,
"OffHeapUnifiedMemory" : 0,
"DirectPoolMemory" : 0,
"MappedPoolMemory" : 0,
"ProcessTreeJVMVMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
"ProcessTreePythonVMemory" : 0,
"ProcessTreePythonRSSMemory" : 0,
"ProcessTreeOtherVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
"MinorGCCount" : 0,
"MinorGCTime" : 0,
"MajorGCCount" : 0,
"MajorGCTime" : 0
}
```
, and the following at the stage level:
```JSON
"peakExecutorMetrics" : {
"JVMHeapMemory" : 213367864,
"JVMOffHeapMemory" : 189011656,
"OnHeapExecutionMemory" : 0,
"OffHeapExecutionMemory" : 0,
"OnHeapStorageMemory" : 2133349,
"OffHeapStorageMemory" : 0,
"OnHeapUnifiedMemory" : 2133349,
"OffHeapUnifiedMemory" : 0,
"DirectPoolMemory" : 282024,
"MappedPoolMemory" : 0,
"ProcessTreeJVMVMemory" : 0,
"ProcessTreeJVMRSSMemory" : 0,
"ProcessTreePythonVMemory" : 0,
"ProcessTreePythonRSSMemory" : 0,
"ProcessTreeOtherVMemory" : 0,
"ProcessTreeOtherRSSMemory" : 0,
"MinorGCCount" : 13,
"MinorGCTime" : 115,
"MajorGCCount" : 4,
"MajorGCTime" : 339
}
```
### How was this patch tested?
Added tests.
Closes #29020 from imback82/metrics.
Lead-authored-by: Terry Kim <yuminkim@gmail.com>
Co-authored-by: edwinalu <edwina.lu@gmail.com>
Signed-off-by: Gengliang Wang <gengliang.wang@databricks.com>
171b7d5d71