2777ed3948
I was wondering what else common sources available until search the source code. Maybe better to make this clear. Author: Yijie Shen <henry.yijieshen@gmail.com> Closes #6641 from yijieshen/patch-1 and squashes the following commits: b5b99b4 [Yijie Shen] Make it clear that JvmSource is the only available additional source currently f23140c [Yijie Shen] [DOC][Minor]Specify the common sources available for collecting
150 lines
5.7 KiB
Plaintext
150 lines
5.7 KiB
Plaintext
# syntax: [instance].sink|source.[name].[options]=[value]
|
|
|
|
# This file configures Spark's internal metrics system. The metrics system is
|
|
# divided into instances which correspond to internal components.
|
|
# Each instance can be configured to report its metrics to one or more sinks.
|
|
# Accepted values for [instance] are "master", "worker", "executor", "driver",
|
|
# and "applications". A wildcard "*" can be used as an instance name, in
|
|
# which case all instances will inherit the supplied property.
|
|
#
|
|
# Within an instance, a "source" specifies a particular set of grouped metrics.
|
|
# there are two kinds of sources:
|
|
# 1. Spark internal sources, like MasterSource, WorkerSource, etc, which will
|
|
# collect a Spark component's internal state. Each instance is paired with a
|
|
# Spark source that is added automatically.
|
|
# 2. Common sources, like JvmSource, which will collect low level state.
|
|
# These can be added through configuration options and are then loaded
|
|
# using reflection.
|
|
#
|
|
# A "sink" specifies where metrics are delivered to. Each instance can be
|
|
# assigned one or more sinks.
|
|
#
|
|
# The sink|source field specifies whether the property relates to a sink or
|
|
# source.
|
|
#
|
|
# The [name] field specifies the name of source or sink.
|
|
#
|
|
# The [options] field is the specific property of this source or sink. The
|
|
# source or sink is responsible for parsing this property.
|
|
#
|
|
# Notes:
|
|
# 1. To add a new sink, set the "class" option to a fully qualified class
|
|
# name (see examples below).
|
|
# 2. Some sinks involve a polling period. The minimum allowed polling period
|
|
# is 1 second.
|
|
# 3. Wildcard properties can be overridden by more specific properties.
|
|
# For example, master.sink.console.period takes precedence over
|
|
# *.sink.console.period.
|
|
# 4. A metrics specific configuration
|
|
# "spark.metrics.conf=${SPARK_HOME}/conf/metrics.properties" should be
|
|
# added to Java properties using -Dspark.metrics.conf=xxx if you want to
|
|
# customize metrics system. You can also put the file in ${SPARK_HOME}/conf
|
|
# and it will be loaded automatically.
|
|
# 5. MetricsServlet is added by default as a sink in master, worker and client
|
|
# driver, you can send http request "/metrics/json" to get a snapshot of all the
|
|
# registered metrics in json format. For master, requests "/metrics/master/json" and
|
|
# "/metrics/applications/json" can be sent seperately to get metrics snapshot of
|
|
# instance master and applications. MetricsServlet may not be configured by self.
|
|
#
|
|
|
|
## List of available common sources and their properties.
|
|
|
|
# org.apache.spark.metrics.source.JvmSource
|
|
# Note: Currently, JvmSource is the only available common source
|
|
# to add additionaly to an instance, to enable this,
|
|
# set the "class" option to its fully qulified class name (see examples below)
|
|
|
|
## List of available sinks and their properties.
|
|
|
|
# org.apache.spark.metrics.sink.ConsoleSink
|
|
# Name: Default: Description:
|
|
# period 10 Poll period
|
|
# unit seconds Units of poll period
|
|
|
|
# org.apache.spark.metrics.sink.CSVSink
|
|
# Name: Default: Description:
|
|
# period 10 Poll period
|
|
# unit seconds Units of poll period
|
|
# directory /tmp Where to store CSV files
|
|
|
|
# org.apache.spark.metrics.sink.GangliaSink
|
|
# Name: Default: Description:
|
|
# host NONE Hostname or multicast group of Ganglia server
|
|
# port NONE Port of Ganglia server(s)
|
|
# period 10 Poll period
|
|
# unit seconds Units of poll period
|
|
# ttl 1 TTL of messages sent by Ganglia
|
|
# mode multicast Ganglia network mode ('unicast' or 'multicast')
|
|
|
|
# org.apache.spark.metrics.sink.JmxSink
|
|
|
|
# org.apache.spark.metrics.sink.MetricsServlet
|
|
# Name: Default: Description:
|
|
# path VARIES* Path prefix from the web server root
|
|
# sample false Whether to show entire set of samples for histograms ('false' or 'true')
|
|
#
|
|
# * Default path is /metrics/json for all instances except the master. The master has two paths:
|
|
# /metrics/applications/json # App information
|
|
# /metrics/master/json # Master information
|
|
|
|
# org.apache.spark.metrics.sink.GraphiteSink
|
|
# Name: Default: Description:
|
|
# host NONE Hostname of Graphite server
|
|
# port NONE Port of Graphite server
|
|
# period 10 Poll period
|
|
# unit seconds Units of poll period
|
|
# prefix EMPTY STRING Prefix to prepend to metric name
|
|
# protocol tcp Protocol ("tcp" or "udp") to use
|
|
|
|
## Examples
|
|
# Enable JmxSink for all instances by class name
|
|
#*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
|
|
|
|
# Enable ConsoleSink for all instances by class name
|
|
#*.sink.console.class=org.apache.spark.metrics.sink.ConsoleSink
|
|
|
|
# Polling period for ConsoleSink
|
|
#*.sink.console.period=10
|
|
|
|
#*.sink.console.unit=seconds
|
|
|
|
# Master instance overlap polling period
|
|
#master.sink.console.period=15
|
|
|
|
#master.sink.console.unit=seconds
|
|
|
|
# Enable CsvSink for all instances
|
|
#*.sink.csv.class=org.apache.spark.metrics.sink.CsvSink
|
|
|
|
# Polling period for CsvSink
|
|
#*.sink.csv.period=1
|
|
|
|
#*.sink.csv.unit=minutes
|
|
|
|
# Polling directory for CsvSink
|
|
#*.sink.csv.directory=/tmp/
|
|
|
|
# Worker instance overlap polling period
|
|
#worker.sink.csv.period=10
|
|
|
|
#worker.sink.csv.unit=minutes
|
|
|
|
# Enable Slf4jSink for all instances by class name
|
|
#*.sink.slf4j.class=org.apache.spark.metrics.sink.Slf4jSink
|
|
|
|
# Polling period for Slf4JSink
|
|
#*.sink.slf4j.period=1
|
|
|
|
#*.sink.slf4j.unit=minutes
|
|
|
|
|
|
# Enable jvm source for instance master, worker, driver and executor
|
|
#master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
|
|
|
|
#worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
|
|
|
|
#driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
|
|
|
|
#executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
|
|
|