ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-7598] [DEPLOY] Add aliveWorkers metrics in Master

Browse source 
In Spark Standalone setup, when some workers are DEAD, they will stay in master worker list for a while.
master.workers metrics for master is only showing the total number of workers, we need to monitor how many real ALIVE workers are there to ensure the cluster is healthy.

Author: Rex Xiong <pengx@microsoft.com>

Closes #6117 from twilightgod/add-aliveWorker-metrics and squashes the following commits:

6be69a5 [Rex Xiong] Fix comment for aliveWorkers metrics
a882f39 [Rex Xiong] Fix style for aliveWorkers metrics
38ce955 [Rex Xiong] Add aliveWorkers metrics in Master

(cherry picked from commit 93dbb3ad83)
Signed-off-by: Andrew Or <andrew@databricks.com>
This commit is contained in:
Rex Xiong 2015-05-14 16:55:31 -07:00 committed by Andrew Or
parent fceaffc49b
commit 894214f9ea
1 changed files with 5 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
core/src/main/scala/org/apache/spark/deploy/master/MasterSource.scala
View file

@ -30,6 +30,11 @@ private[spark] class MasterSource(val master: Master) extends Source {
override def getValue: Int = master.workers.size override def getValue: Int = master.workers.size
}) })
// Gauge for alive worker numbers in cluster
metricRegistry.register(MetricRegistry.name("aliveWorkers"), new Gauge[Int]{
override def getValue: Int = master.workers.filter(_.state == WorkerState.ALIVE).size
})
// Gauge for application numbers in cluster // Gauge for application numbers in cluster
metricRegistry.register(MetricRegistry.name("apps"), new Gauge[Int] { metricRegistry.register(MetricRegistry.name("apps"), new Gauge[Int] {
override def getValue: Int = master.apps.size override def getValue: Int = master.apps.size