Exposes data available in the UI as json over http. Key points: * new endpoints, handled independently of existing XyzPage classes. Root entrypoint is `JsonRootResource` * Uses jersey + jackson for routing & converting POJOs into json * tests against known results in `HistoryServerSuite` * also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages. Author: Imran Rashid <irashid@cloudera.com> Closes #4435 from squito/SPARK-3454 and squashes the following commits: da1e35f [Imran Rashid] typos etc. 5e78b4f [Imran Rashid] fix rendering problems 5ae02ad [Imran Rashid] Merge branch 'master' into SPARK-3454 f016182 [Imran Rashid] change all constructors json-pojo class constructors to be private[spark] to protect us from mima-false-positives if we add fields 3347b72 [Imran Rashid] mark EnumUtil as @Private ec140a2 [Imran Rashid] create @Private cc1febf [Imran Rashid] add docs on the metrics-as-json api cbaf287 [Imran Rashid] Merge branch 'master' into SPARK-3454 56db31e [Imran Rashid] update tests for mulit-attempt 7f3bc4e [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" 67008b4 [Imran Rashid] rats 9e51400 [Imran Rashid] style c9bae1c [Imran Rashid] handle multiple attempts per app b87cd63 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 188762c [Imran Rashid] multi-attempt 2af11e5 [Imran Rashid] Merge branch 'master' into SPARK-3454 befff0c [Imran Rashid] review feedback 14ac3ed [Imran Rashid] jersey-core needs to be explicit; move version & scope to parent pom.xml f90680e [Imran Rashid] Merge branch 'master' into SPARK-3454 dc8a7fe [Imran Rashid] style, fix errant comments acb7ef6 [Imran Rashid] fix indentation 7bf1811 [Imran Rashid] move MetricHelper so mima doesnt think its exposed; comments 9d889d6 [Imran Rashid] undo some unnecessary changes f48a7b0 [Imran Rashid] docs 52bbae8 [Imran Rashid] StorageListener & StorageStatusListener needs to synchronize internally to be thread-safe 31c79ce [Imran Rashid] asm no longer needed for SPARK_PREPEND_CLASSES b2f8b91 [Imran Rashid] @DeveloperApi 2e19be2 [Imran Rashid] lazily convert ApplicationInfo to avoid memory overhead ba3d9d2 [Imran Rashid] upper case enums 39ac29c [Imran Rashid] move EnumUtil d2bde77 [Imran Rashid] update error handling & scoping 4a234d3 [Imran Rashid] avoid jersey-media-json-jackson b/c of potential version conflicts a157a2f [Imran Rashid] style 7bd4d15 [Imran Rashid] delete security test, since it doesnt do anything a325563 [Imran Rashid] style a9c5cf1 [Imran Rashid] undo changes superceeded by master 0c6f968 [Imran Rashid] update deps 1ed0d07 [Imran Rashid] Merge branch 'master' into SPARK-3454 4c92af6 [Imran Rashid] style f2e63ad [Imran Rashid] Merge branch 'master' into SPARK-3454 c22b11f [Imran Rashid] fix compile error 9ea682c [Imran Rashid] go back to good ol' java enums cf86175 [Imran Rashid] style d493b38 [Imran Rashid] Merge branch 'master' into SPARK-3454 f05ae89 [Imran Rashid] add in ExecutorSummaryInfo for MiMa :( 101a698 [Imran Rashid] style d2ef58d [Imran Rashid] revert changes that had HistoryServer refresh the application listing more often b136e39b [Imran Rashid] Revert "add sbt-revolved plugin, to make it easier to start & stop http servers in sbt" e031719 [Imran Rashid] fixes from review 1f53a66 [Imran Rashid] style b4a7863 [Imran Rashid] fix compile error 2c8b7ee [Imran Rashid] rats 1578a4a [Imran Rashid] doc 674f8dc [Imran Rashid] more explicit about total numbers of jobs & stages vs. number retained 9922be0 [Imran Rashid] Merge branch 'master' into stage_distributions f5a5196 [Imran Rashid] undo removal of renderJson from MasterPage, since there is no substitute yet db61211 [Imran Rashid] get JobProgressListener directly from UI fdfc181 [Imran Rashid] stage/taskList 63eb4a6 [Imran Rashid] tests for taskSummary ad27de8 [Imran Rashid] error handling on quantile values b2efcaf [Imran Rashid] cleanup, combine stage-related paths into one resource aaba896 [Imran Rashid] wire up task summary a4b1397 [Imran Rashid] stage metric distributions e48ba32 [Imran Rashid] rename eaf3bbb [Imran Rashid] style 25cd894 [Imran Rashid] if only given day, assume GMT 51eaedb [Imran Rashid] more visibility fixes 9f28b7e [Imran Rashid] ack, more cleanup 99764e1 [Imran Rashid] Merge branch 'SPARK-3454_w_jersey' into SPARK-3454 a61a43c [Imran Rashid] oops, remove accidental checkin a066055 [Imran Rashid] set visibility on a lot of classes 1f361c8 [Imran Rashid] update rat-excludes 0be5120 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 2382bef [Imran Rashid] switch to using new "enum" fef6605 [Imran Rashid] some utils for working w/ new "enum" format dbfc7bf [Imran Rashid] style b86bcb0 [Imran Rashid] update test to look at one stage attempt 5f9df24 [Imran Rashid] style 7fd156a [Imran Rashid] refactor jsonDiff to avoid code duplication 73f1378 [Imran Rashid] test json; also add test cases for cleaned stages & jobs 97d411f [Imran Rashid] json endpoint for one job 0c96147 [Imran Rashid] better error msgs for bad stageId vs bad attemptId dddbd29 [Imran Rashid] stages have attempt; jobs are sorted; resource for all attempts for one stage 190c17a [Imran Rashid] StagePage should distinguish no task data, from unknown stage 84cd497 [Imran Rashid] AllJobsPage should still report correct completed & failed job count, even if some have been cleaned, to make it consistent w/ AllStagesPage 36e4062 [Imran Rashid] SparkUI needs to know about startTime, so it can list its own applicationInfo b4c75ed [Imran Rashid] fix merge conflicts; need to widen visibility in a few cases e91750a [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 56d2fc7 [Imran Rashid] jersey needs asm for SPARK_PREPEND_CLASSES to work f7df095 [Imran Rashid] add test for accumulables, and discover that I need update after all 9c0c125 [Imran Rashid] add accumulableInfo 00e9cc5 [Imran Rashid] more style 3377e61 [Imran Rashid] scaladoc d05f7a9 [Imran Rashid] dont use case classes for status api POJOs, since they have binary compatibility issues 654cecf [Imran Rashid] move all the status api POJOs to one file b86e2b0 [Imran Rashid] style 18a8c45 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey 5598f19 [Imran Rashid] delete some unnecessary code, more to go 56edce0 [Imran Rashid] style 017c755 [Imran Rashid] add in metrics now available 1b78cb7 [Imran Rashid] fix some import ordering 0dc3ea7 [Imran Rashid] if app isnt found, reload apps from FS before giving up c7d884f [Imran Rashid] fix merge conflicts 0c12b50 [Imran Rashid] Merge branch 'master' into SPARK-3454_w_jersey b6a96a8 [Imran Rashid] compare json by AST, not stringcd37845
[Imran Rashid] switch to using java.util.Dates for times a4ab5aa [Imran Rashid] add in explicit dependency on jersey 1.9 -- maven wasn't happy before this 4fdc39f [Imran Rashid] refactor case insensitive enum parsing cba1ef6 [Imran Rashid] add security (maybe?) for metrics json f0264a7 [Imran Rashid] switch to using jersey for metrics json bceb3a9 [Imran Rashid] set http response code on error, some testing e0356b6 [Imran Rashid] put new test expectation files in rat excludes (is this OK?) b252e7a [Imran Rashid] small cleanup of accidental changes d1a8c92 [Imran Rashid] add sbt-revolved plugin, to make it easier to start & stop http servers in sbt 4b398d0 [Imran Rashid] expose UI data as json in new endpoints (cherry picked from commitd49735800d
) Signed-off-by: Imran Rashid <irashid@cloudera.com>
13 KiB
layout | title | description |
---|---|---|
global | Monitoring and Instrumentation | Monitoring, metrics, and instrumentation guide for Spark SPARK_VERSION_SHORT |
There are several ways to monitor Spark applications: web UIs, metrics, and external instrumentation.
Web Interfaces
Every SparkContext launches a web UI, by default on port 4040, that displays useful information about the application. This includes:
- A list of scheduler stages and tasks
- A summary of RDD sizes and memory usage
- Environmental information.
- Information about the running executors
You can access this interface by simply opening http://<driver-node>:4040
in a web browser.
If multiple SparkContexts are running on the same host, they will bind to successive ports
beginning with 4040 (4041, 4042, etc).
Note that this information is only available for the duration of the application by default.
To view the web UI after the fact, set spark.eventLog.enabled
to true before starting the
application. This configures Spark to log Spark events that encode the information displayed
in the UI to persisted storage.
Viewing After the Fact
Spark's Standalone Mode cluster manager also has its own web UI. If an application has logged events over the course of its lifetime, then the Standalone master's web UI will automatically re-render the application's UI after the application has finished.
If Spark is run on Mesos or YARN, it is still possible to reconstruct the UI of a finished application through Spark's history server, provided that the application's event logs exist. You can start the history server by executing:
./sbin/start-history-server.sh
When using the file-system provider class (see spark.history.provider below), the base logging
directory must be supplied in the spark.history.fs.logDirectory
configuration option,
and should contain sub-directories that each represents an application's event logs. This creates a
web interface at http://<server-url>:18080
by default. The history server can be configured as
follows:
Environment Variable | Meaning |
---|---|
SPARK_DAEMON_MEMORY |
Memory to allocate to the history server (default: 512m). |
SPARK_DAEMON_JAVA_OPTS |
JVM options for the history server (default: none). |
SPARK_PUBLIC_DNS |
The public address for the history server. If this is not set, links to application history may use the internal address of the server, resulting in broken links (default: none). |
SPARK_HISTORY_OPTS |
spark.history.* configuration options for the history server (default: none).
|
Property Name | Default | Meaning |
---|---|---|
spark.history.provider | org.apache.spark.deploy.history.FsHistoryProvider | Name of the class implementing the application history backend. Currently there is only one implementation, provided by Spark, which looks for application logs stored in the file system. |
spark.history.fs.logDirectory | file:/tmp/spark-events | Directory that contains application event logs to be loaded by the history server |
spark.history.fs.update.interval | 10s | The period at which information displayed by this history server is updated. Each update checks for any changes made to the event logs in persisted storage. |
spark.history.retainedApplications | 50 | The number of application UIs to retain. If this cap is exceeded, then the oldest applications will be removed. |
spark.history.ui.port | 18080 | The port to which the web interface of the history server binds. |
spark.history.kerberos.enabled | false |
Indicates whether the history server should use kerberos to login. This is useful
if the history server is accessing HDFS files on a secure Hadoop cluster. If this is
true, it uses the configs spark.history.kerberos.principal and
spark.history.kerberos.keytab .
|
spark.history.kerberos.principal | (none) | Kerberos principal name for the History Server. |
spark.history.kerberos.keytab | (none) | Location of the kerberos keytab file for the History Server. |
spark.history.ui.acls.enable | false |
Specifies whether acls should be checked to authorize users viewing the applications.
If enabled, access control checks are made regardless of what the individual application had
set for spark.ui.acls.enable when the application was run. The application owner
will always have authorization to view their own application and any users specified via
spark.ui.view.acls when the application was run will also have authorization
to view that application.
If disabled, no access control checks are made.
|
spark.history.fs.cleaner.enabled | false | Specifies whether the History Server should periodically clean up event logs from storage. |
spark.history.fs.cleaner.interval | 1d | How often the job history cleaner checks for files to delete. Files are only deleted if they are older than spark.history.fs.cleaner.maxAge. |
spark.history.fs.cleaner.maxAge | 7d | Job history files older than this will be deleted when the history cleaner runs. |
Note that in all of these UIs, the tables are sortable by clicking their headers, making it easy to identify slow tasks, data skew, etc.
Note that the history server only displays completed Spark jobs. One way to signal the completion of a Spark job is to stop the Spark Context explicitly (sc.stop()
), or in Python using the with SparkContext() as sc:
to handle the Spark Context setup and tear down, and still show the job history on the UI.
REST API
In addition to viewing the metrics in the UI, they are also available as JSON. This gives developers
an easy way to create new visualizations and monitoring tools for Spark. The JSON is available for
both running applications, and in the history server. The endpoints are mounted at /json/v1
. Eg.,
for the history server, they would typically be accessible at http://<server-url>:18080/json/v1
, and
for a running application, at http://localhost:4040/json/v1
.
Endpoint | Meaning |
---|---|
/applications |
A list of all applications |
/applications/[app-id]/jobs |
A list of all jobs for a given application |
/applications/[app-id]/jobs/[job-id] |
Details for the given job |
/applications/[app-id]/stages |
A list of all stages for a given application |
/applications/[app-id]/stages/[stage-id] |
A list of all attempts for the given stage |
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id] |
Details for the given stage attempt |
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary |
Summary metrics of all tasks in the given stage attempt |
/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskList |
A list of all tasks for the given stage attempt |
/applications/[app-id]/executors |
A list of all executors for the given application |
/applications/[app-id]/storage/rdd |
A list of stored RDDs for the given application |
/applications/[app-id]/storage/rdd/[rdd-id] |
Details for the storage status of a given RDD |
When running on Yarn, each application has multiple attempts, so [app-id]
is actually
[app-id]/[attempt-id]
in all cases.
These endpoints have been strongly versioned to make it easier to develop applications on top. In particular, Spark guarantees:
- Endpoints will never be removed from one version
- Individual fields will never be removed for any given endpoint
- New endpoints may be added
- New fields may be added to existing endpoints
- New versions of the api may be added in the future at a separate endpoint (eg.,
json/v2
). New versions are not required to be backwards compatible. - Api versions may be dropped, but only after at least one minor release of co-existing with a new api version
Note that even when examining the UI of a running applications, the applications/[app-id]
portion is
still required, though there is only one application available. Eg. to see the list of jobs for the
running app, you would go to http://localhost:4040/json/v1/applications/[app-id]/jobs
. This is to
keep the paths consistent in both modes.
Metrics
Spark has a configurable metrics system based on the
Coda Hale Metrics Library.
This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV
files. The metrics system is configured via a configuration file that Spark expects to be present
at $SPARK_HOME/conf/metrics.properties
. A custom file location can be specified via the
spark.metrics.conf
configuration property.
Spark's metrics are decoupled into different
instances corresponding to Spark components. Within each instance, you can configure a
set of sinks to which metrics are reported. The following instances are currently supported:
master
: The Spark standalone master process.applications
: A component within the master which reports on various applications.worker
: A Spark standalone worker process.executor
: A Spark executor.driver
: The Spark driver process (the process in which your SparkContext is created).
Each instance can report to zero or more sinks. Sinks are contained in the
org.apache.spark.metrics.sink
package:
ConsoleSink
: Logs metrics information to the console.CSVSink
: Exports metrics data to CSV files at regular intervals.JmxSink
: Registers metrics for viewing in a JMX console.MetricsServlet
: Adds a servlet within the existing Spark UI to serve metrics data as JSON data.GraphiteSink
: Sends metrics to a Graphite node.Slf4jSink
: Sends metrics to slf4j as log entries.
Spark also supports a Ganglia sink which is not included in the default build due to licensing restrictions:
GangliaSink
: Sends metrics to a Ganglia node or multicast group.
To install the GangliaSink
you'll need to perform a custom build of Spark. Note that
by embedding this library you will include LGPL-licensed
code in your Spark package. For sbt users, set the
SPARK_GANGLIA_LGPL
environment variable before building. For Maven users, enable
the -Pspark-ganglia-lgpl
profile. In addition to modifying the cluster's Spark build
user applications will need to link to the spark-ganglia-lgpl
artifact.
The syntax of the metrics configuration file is defined in an example configuration file,
$SPARK_HOME/conf/metrics.properties.template
.
Advanced Instrumentation
Several external tools can be used to help profile the performance of Spark jobs:
- Cluster-wide monitoring tools, such as Ganglia, can provide insight into overall cluster utilization and resource bottlenecks. For instance, a Ganglia dashboard can quickly reveal whether a particular workload is disk bound, network bound, or CPU bound.
- OS profiling tools such as dstat, iostat, and iotop can provide fine-grained profiling on individual nodes.
- JVM utilities such as
jstack
for providing stack traces,jmap
for creating heap-dumps,jstat
for reporting time-series statistics andjconsole
for visually exploring various JVM properties are useful for those comfortable with JVM internals.