2888009d66
### What changes were proposed in this pull request? The Spark metrics system produces many different metrics and not all of them are used at the same time. This proposes to introduce a configuration parameter to allow disabling the registration of metrics in the "static sources" category. ### Why are the changes needed? This allows to reduce the load and clutter on the sink, in the cases when the metrics in question are not needed. The metrics registerd as "static sources" are under the namespaces CodeGenerator and HiveExternalCatalog and can produce a significant amount of data, as they are registered for the driver and executors. ### Does this PR introduce any user-facing change? It introduces a new configuration parameter `spark.metrics.register.static.sources.enabled` ### How was this patch tested? Manually tested. ``` $ cat conf/metrics.properties *.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet *.sink.prometheusServlet.path=/metrics/prometheus master.sink.prometheusServlet.path=/metrics/master/prometheus applications.sink.prometheusServlet.path=/metrics/applications/prometheus $ bin/spark-shell $ curl -s http://localhost:4040/metrics/prometheus/ | grep Hive metrics_local_1573330115306_driver_HiveExternalCatalog_fileCacheHits_Count 0 metrics_local_1573330115306_driver_HiveExternalCatalog_filesDiscovered_Count 0 metrics_local_1573330115306_driver_HiveExternalCatalog_hiveClientCalls_Count 0 metrics_local_1573330115306_driver_HiveExternalCatalog_parallelListingJobCount_Count 0 metrics_local_1573330115306_driver_HiveExternalCatalog_partitionsFetched_Count 0 $ bin/spark-shell --conf spark.metrics.static.sources.enabled=false $ curl -s http://localhost:4040/metrics/prometheus/ | grep Hive ``` Closes #26320 from LucaCanali/addConfigRegisterStaticMetrics. Authored-by: Luca Canali <luca.canali@cern.ch> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||