171b7d5d71
### What changes were proposed in this pull request? Note that this PR is forked from #23340 originally written by edwinalu. This PR proposes to expose the peak executor metrics at the stage level via the REST APIs: * `/applications/<application_id>/stages/`: peak values of executor metrics for **each stage** * `/applications/<application_id>/stages/<stage_id>/< stage_attempt_id >`: peak values of executor metrics for **each executor** for the stage, followed by peak values of executor metrics for the stage ### Why are the changes needed? The stage level peak executor metrics can help better understand your application's resource utilization. ### Does this PR introduce _any_ user-facing change? 1. For the `/applications/<application_id>/stages/` API, you will see the following new info for **each stage**: ```JSON "peakExecutorMetrics" : { "JVMHeapMemory" : 213367864, "JVMOffHeapMemory" : 189011656, "OnHeapExecutionMemory" : 0, "OffHeapExecutionMemory" : 0, "OnHeapStorageMemory" : 2133349, "OffHeapStorageMemory" : 0, "OnHeapUnifiedMemory" : 2133349, "OffHeapUnifiedMemory" : 0, "DirectPoolMemory" : 282024, "MappedPoolMemory" : 0, "ProcessTreeJVMVMemory" : 0, "ProcessTreeJVMRSSMemory" : 0, "ProcessTreePythonVMemory" : 0, "ProcessTreePythonRSSMemory" : 0, "ProcessTreeOtherVMemory" : 0, "ProcessTreeOtherRSSMemory" : 0, "MinorGCCount" : 13, "MinorGCTime" : 115, "MajorGCCount" : 4, "MajorGCTime" : 339 } ``` 2. For the `/applications/<application_id>/stages/<stage_id>/<stage_attempt_id>` API, you will see the following new info for **each executor** under `executorSummary`: ```JSON "peakMemoryMetrics" : { "JVMHeapMemory" : 0, "JVMOffHeapMemory" : 0, "OnHeapExecutionMemory" : 0, "OffHeapExecutionMemory" : 0, "OnHeapStorageMemory" : 0, "OffHeapStorageMemory" : 0, "OnHeapUnifiedMemory" : 0, "OffHeapUnifiedMemory" : 0, "DirectPoolMemory" : 0, "MappedPoolMemory" : 0, "ProcessTreeJVMVMemory" : 0, "ProcessTreeJVMRSSMemory" : 0, "ProcessTreePythonVMemory" : 0, "ProcessTreePythonRSSMemory" : 0, "ProcessTreeOtherVMemory" : 0, "ProcessTreeOtherRSSMemory" : 0, "MinorGCCount" : 0, "MinorGCTime" : 0, "MajorGCCount" : 0, "MajorGCTime" : 0 } ``` , and the following at the stage level: ```JSON "peakExecutorMetrics" : { "JVMHeapMemory" : 213367864, "JVMOffHeapMemory" : 189011656, "OnHeapExecutionMemory" : 0, "OffHeapExecutionMemory" : 0, "OnHeapStorageMemory" : 2133349, "OffHeapStorageMemory" : 0, "OnHeapUnifiedMemory" : 2133349, "OffHeapUnifiedMemory" : 0, "DirectPoolMemory" : 282024, "MappedPoolMemory" : 0, "ProcessTreeJVMVMemory" : 0, "ProcessTreeJVMRSSMemory" : 0, "ProcessTreePythonVMemory" : 0, "ProcessTreePythonRSSMemory" : 0, "ProcessTreeOtherVMemory" : 0, "ProcessTreeOtherRSSMemory" : 0, "MinorGCCount" : 13, "MinorGCTime" : 115, "MajorGCCount" : 4, "MajorGCTime" : 339 } ``` ### How was this patch tested? Added tests. Closes #29020 from imback82/metrics. Lead-authored-by: Terry Kim <yuminkim@gmail.com> Co-authored-by: edwinalu <edwina.lu@gmail.com> Signed-off-by: Gengliang Wang <gengliang.wang@databricks.com>
127 lines
2 KiB
Plaintext
127 lines
2 KiB
Plaintext
target
|
|
cache
|
|
.gitignore
|
|
.gitattributes
|
|
.project
|
|
.classpath
|
|
.mima-excludes
|
|
.generated-mima-excludes
|
|
.generated-mima-class-excludes
|
|
.generated-mima-member-excludes
|
|
.rat-excludes
|
|
.*md
|
|
derby.log
|
|
licenses/*
|
|
licenses-binary/*
|
|
LICENSE
|
|
NOTICE
|
|
TAGS
|
|
RELEASE
|
|
control
|
|
docs
|
|
slaves
|
|
spark-env.cmd
|
|
bootstrap.bundle.min.js
|
|
bootstrap.bundle.min.js.map
|
|
bootstrap.min.css
|
|
bootstrap.min.css.map
|
|
jquery-3.4.1.min.js
|
|
d3.min.js
|
|
dagre-d3.min.js
|
|
graphlib-dot.min.js
|
|
sorttable.js
|
|
vis-timeline-graph2d.min.js
|
|
vis-timeline-graph2d.min.css
|
|
dataTables.bootstrap4.1.10.20.min.css
|
|
dataTables.bootstrap4.1.10.20.min.js
|
|
dataTables.rowsGroup.js
|
|
jquery.blockUI.min.js
|
|
jquery.cookies.2.2.0.min.js
|
|
jquery.dataTables.1.10.20.min.css
|
|
jquery.dataTables.1.10.20.min.js
|
|
jquery.mustache.js
|
|
jsonFormatter.min.css
|
|
jsonFormatter.min.js
|
|
.*avsc
|
|
.*txt
|
|
.*json
|
|
.*data
|
|
.*log
|
|
pyspark-coverage-site/*
|
|
cloudpickle/*
|
|
join.py
|
|
SparkExprTyper.scala
|
|
SparkILoop.scala
|
|
SparkILoopInit.scala
|
|
SparkIMain.scala
|
|
SparkImports.scala
|
|
SparkJLineCompletion.scala
|
|
SparkJLineReader.scala
|
|
SparkMemberHandlers.scala
|
|
SparkReplReporter.scala
|
|
sbt
|
|
sbt-launch-lib.bash
|
|
plugins.sbt
|
|
work
|
|
.*\.q
|
|
.*\.qv
|
|
golden
|
|
test.out/*
|
|
.*iml
|
|
service.properties
|
|
db.lck
|
|
build/*
|
|
dist/*
|
|
.*out
|
|
.*ipr
|
|
.*iws
|
|
logs
|
|
.*scalastyle-output.xml
|
|
.*dependency-reduced-pom.xml
|
|
known_translations
|
|
json_expectation
|
|
app-20180109111548-0000
|
|
app-20161115172038-0000
|
|
app-20161116163331-0000
|
|
application_1516285256255_0012
|
|
application_1553914137147_0018
|
|
application_1555004656427_0144
|
|
stat
|
|
local-1422981759269
|
|
local-1422981780767
|
|
local-1425081759269
|
|
local-1426533911241
|
|
local-1426633911242
|
|
local-1430917381534
|
|
local-1430917381535_1
|
|
local-1430917381535_2
|
|
DESCRIPTION
|
|
NAMESPACE
|
|
test_support/*
|
|
.*Rd
|
|
help/*
|
|
html/*
|
|
INDEX
|
|
.lintr
|
|
gen-java.*
|
|
.*avpr
|
|
.*parquet
|
|
spark-deps-.*
|
|
.*csv
|
|
.*tsv
|
|
.*\.sql
|
|
.Rbuildignore
|
|
META-INF/*
|
|
spark-warehouse
|
|
structured-streaming/*
|
|
kafka-source-initial-offset-version-2.1.0.bin
|
|
kafka-source-initial-offset-future-version.bin
|
|
announce.tmpl
|
|
vote.tmpl
|
|
SessionManager.java
|
|
SessionHandler.java
|
|
GangliaReporter.java
|
|
application_1578436911597_0052
|
|
config.properties
|
|
app-20200706201101-0003
|