90c77ea313
This adds the entire memory used by spark’s executor (as measured by procfs) to the executor metrics. The memory usage is collected from the entire process tree under the executor. The metrics are subdivided into memory used by java, by python, and by other processes, to aid users in diagnosing the source of high memory usage. The additional metrics are sent to the driver in heartbeats, using the mechanism introduced by SPARK-23429. This also slightly extends that approach to allow one ExecutorMetricType to collect multiple metrics. Added unit tests and also tested on a live cluster. Closes #22612 from rezasafi/ptreememory2. Authored-by: Reza Safi <rezasafi@cloudera.com> Signed-off-by: Imran Rashid <irashid@cloudera.com>
117 lines
1.7 KiB
Plaintext
117 lines
1.7 KiB
Plaintext
target
|
|
cache
|
|
.gitignore
|
|
.gitattributes
|
|
.project
|
|
.classpath
|
|
.mima-excludes
|
|
.generated-mima-excludes
|
|
.generated-mima-class-excludes
|
|
.generated-mima-member-excludes
|
|
.rat-excludes
|
|
.*md
|
|
derby.log
|
|
licenses/*
|
|
licenses-binary/*
|
|
LICENSE
|
|
NOTICE
|
|
TAGS
|
|
RELEASE
|
|
control
|
|
docs
|
|
slaves
|
|
spark-env.cmd
|
|
bootstrap-tooltip.js
|
|
jquery-1.11.1.min.js
|
|
d3.min.js
|
|
dagre-d3.min.js
|
|
graphlib-dot.min.js
|
|
sorttable.js
|
|
vis.min.js
|
|
vis.min.css
|
|
dataTables.bootstrap.css
|
|
dataTables.bootstrap.min.js
|
|
dataTables.rowsGroup.js
|
|
jquery.blockUI.min.js
|
|
jquery.cookies.2.2.0.min.js
|
|
jquery.dataTables.1.10.4.min.css
|
|
jquery.dataTables.1.10.4.min.js
|
|
jquery.mustache.js
|
|
jsonFormatter.min.css
|
|
jsonFormatter.min.js
|
|
.*avsc
|
|
.*txt
|
|
.*json
|
|
.*data
|
|
.*log
|
|
cloudpickle.py
|
|
heapq3.py
|
|
join.py
|
|
SparkExprTyper.scala
|
|
SparkILoop.scala
|
|
SparkILoopInit.scala
|
|
SparkIMain.scala
|
|
SparkImports.scala
|
|
SparkJLineCompletion.scala
|
|
SparkJLineReader.scala
|
|
SparkMemberHandlers.scala
|
|
SparkReplReporter.scala
|
|
sbt
|
|
sbt-launch-lib.bash
|
|
plugins.sbt
|
|
work
|
|
.*\.q
|
|
.*\.qv
|
|
golden
|
|
test.out/*
|
|
.*iml
|
|
service.properties
|
|
db.lck
|
|
build/*
|
|
dist/*
|
|
.*out
|
|
.*ipr
|
|
.*iws
|
|
logs
|
|
.*scalastyle-output.xml
|
|
.*dependency-reduced-pom.xml
|
|
known_translations
|
|
json_expectation
|
|
app-20180109111548-0000
|
|
app-20161115172038-0000
|
|
app-20161116163331-0000
|
|
application_1516285256255_0012
|
|
application_1506645932520_24630151
|
|
application_1538416563558_0014
|
|
stat
|
|
local-1422981759269
|
|
local-1422981780767
|
|
local-1425081759269
|
|
local-1426533911241
|
|
local-1426633911242
|
|
local-1430917381534
|
|
local-1430917381535_1
|
|
local-1430917381535_2
|
|
DESCRIPTION
|
|
NAMESPACE
|
|
test_support/*
|
|
.*Rd
|
|
help/*
|
|
html/*
|
|
INDEX
|
|
.lintr
|
|
gen-java.*
|
|
.*avpr
|
|
.*parquet
|
|
spark-deps-.*
|
|
.*csv
|
|
.*tsv
|
|
.*\.sql
|
|
.Rbuildignore
|
|
META-INF/*
|
|
spark-warehouse
|
|
structured-streaming/*
|
|
kafka-source-initial-offset-version-2.1.0.bin
|
|
kafka-source-initial-offset-future-version.bin
|
|
vote.tmpl
|