73747ecb97
### What changes were proposed in this pull request? Currently there are no speculation metrics available for Spark either at application/job/stage level. This PR is to add some basic speculation metrics for a stage when speculation execution is enabled. This is similar to the existing stage level metrics tracking numTotal (total number of speculated tasks), numCompleted (total number of successful speculated tasks), numFailed (total number of failed speculated tasks), numKilled (total number of killed speculated tasks) etc. With this new set of metrics, it helps further understanding speculative execution feature in the context of the application and also helps in further tuning the speculative execution config knobs. Screenshot of Spark UI with speculation summary:  Screenshot of Spark UI with API output:  ### Why are the changes needed? Additional metrics for speculative execution. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests added and also deployed in our internal platform for quite some time now. Lead-authored by: Venkata krishnan Sowrirajan <vsowrirajanlinkedin.com> Co-authored by: Ron Hu <rhulinkedin.com> Co-authored by: Thejdeep Gudivada <tgudivadalinkedin.com> Closes #33253 from venkata91/speculation-metrics. Authored-by: Venkata krishnan Sowrirajan <vsowrirajan@linkedin.com> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com> |
||
|---|---|---|
| .. | ||
| ansible-for-test-node | ||
| create-release | ||
| deps | ||
| sparktestsupport | ||
| tests | ||
| .gitignore | ||
| .rat-excludes | ||
| .scalafmt.conf | ||
| appveyor-guide.md | ||
| appveyor-install-dependencies.ps1 | ||
| change-scala-version.sh | ||
| check-license | ||
| checkstyle-suppressions.xml | ||
| checkstyle.xml | ||
| eslint.json | ||
| github_jira_sync.py | ||
| lint-java | ||
| lint-js | ||
| lint-python | ||
| lint-r | ||
| lint-r.R | ||
| lint-scala | ||
| make-distribution.sh | ||
| merge_spark_pr.py | ||
| mima | ||
| package-lock.json | ||
| package.json | ||
| pip-sanity-check.py | ||
| README.md | ||
| reformat-python | ||
| requirements.txt | ||
| run-pip-tests | ||
| run-tests | ||
| run-tests-jenkins | ||
| run-tests-jenkins.py | ||
| run-tests.py | ||
| sbt-checkstyle | ||
| scalafmt | ||
| scalastyle | ||
| test-dependencies.sh | ||
| tox.ini | ||
Spark Developer Scripts
This directory contains scripts useful to developers when packaging, testing, or committing to Spark.
Many of these scripts require Apache credentials to work correctly.