7fe0f3f2b4
I sometimes trouble-shoot and analyse the cause of long time spending job.
At the time, I find the stages which spends long time or fails, then I find the tasks which spends long time or fails, next I analyse the proportion of each phase in a task.
Another case, I find executors which spends long time for running a task and analyse the details of a task.
In such situation, I think it's helpful to visualize timeline for each application, job, task and the details of proportion of activity for each task.
I added 3 timeline features to existing Web UI.
[Application Timeline View]
This view shows following things.
* When each executor was added/removed and the reason why it's removed.
* When each job was started/finished.
* Status of each job.
[Stage Timeline View]
Similar to Application Timeline View, this view shows following things.
* When each executor was added/removed and the reason why it's removed.
* When each job was started/finished.
* Status of each stage.
[Task Assignment Timeline View]
This view shows following things.
* When each task started/finished
* How long each task spent and the proportion.
* Status of each task.
* Where each task ran on.
All the view above is zoomable by mouse wheel action and scrollable by drag action.
Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Closes #2342 from sarutak/timeline-viewer-feature and squashes the following commits:
11fe67d [Kousuke Saruta] Fixed conflict
79ac03d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
a91abd3 [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature
ef34a5b [Kousuke Saruta] Implement tooltip using bootstrap
b09d0c5 [Kousuke Saruta] Move `stroke` and `fill` attribute of rect elements to css
d3c63c8 [Kousuke Saruta] Fixed a little bit bugs
a36291b [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature
28714b6 [Kousuke Saruta] Fixed highlight issue
0dc4278 [Kousuke Saruta] Addressed most of Patrics's feedbacks
8110acf [Kousuke Saruta] Added scroll limit to Job timeline
974a64a [Kousuke Saruta] Removed unused function
ee7a7f0 [Kousuke Saruta] Refactored
6a91872 [Kousuke Saruta] Temporary commit
6693f34 [Kousuke Saruta] Added link to job/stage box in the timeline in order to move to corresponding row when we click
8f88222 [Kousuke Saruta] Added job/stage description
aeed4b1 [Kousuke Saruta] Removed stage timeline
fc1696c [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature
999ccd4 [Kousuke Saruta] Improved scalability
0fc6a31 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
19815ae [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
68b7540 [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature
52b5f0b [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
dec85db [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
fcdab7d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
dab7cc1 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
09cce97 [Kousuke Saruta] Cleanuped
16f82cf [Kousuke Saruta] Cleanuped
9fb522e [Kousuke Saruta] Cleanuped
d05f2c2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
e85e9aa [Kousuke Saruta] Cleanup: Added TimelineViewUtils.scala
a76e569 [Kousuke Saruta] Removed unused setting in timeline-view.css
5ce1b21 [Kousuke Saruta] Added vis.min.js, vis.min.css and vis.map to .rat-exclude
082f709 [Kousuke Saruta] Added Timeline-View feature for Applications, Jobs and Stages
75 lines
1 KiB
Plaintext
75 lines
1 KiB
Plaintext
target
|
|
cache
|
|
.gitignore
|
|
.gitattributes
|
|
.project
|
|
.classpath
|
|
.mima-excludes
|
|
.generated-mima-excludes
|
|
.generated-mima-class-excludes
|
|
.generated-mima-member-excludes
|
|
.rat-excludes
|
|
.*md
|
|
derby.log
|
|
TAGS
|
|
RELEASE
|
|
control
|
|
docs
|
|
fairscheduler.xml.template
|
|
spark-defaults.conf.template
|
|
log4j.properties
|
|
log4j.properties.template
|
|
metrics.properties
|
|
metrics.properties.template
|
|
slaves
|
|
slaves.template
|
|
spark-env.sh
|
|
spark-env.cmd
|
|
spark-env.sh.template
|
|
log4j-defaults.properties
|
|
bootstrap-tooltip.js
|
|
jquery-1.11.1.min.js
|
|
sorttable.js
|
|
vis.min.js
|
|
vis.min.css
|
|
vis.map
|
|
.*avsc
|
|
.*txt
|
|
.*json
|
|
.*data
|
|
.*log
|
|
cloudpickle.py
|
|
heapq3.py
|
|
join.py
|
|
SparkExprTyper.scala
|
|
SparkILoop.scala
|
|
SparkILoopInit.scala
|
|
SparkIMain.scala
|
|
SparkImports.scala
|
|
SparkJLineCompletion.scala
|
|
SparkJLineReader.scala
|
|
SparkMemberHandlers.scala
|
|
SparkReplReporter.scala
|
|
sbt
|
|
sbt-launch-lib.bash
|
|
plugins.sbt
|
|
work
|
|
.*\.q
|
|
.*\.qv
|
|
golden
|
|
test.out/*
|
|
.*iml
|
|
service.properties
|
|
db.lck
|
|
build/*
|
|
dist/*
|
|
.*out
|
|
.*ipr
|
|
.*iws
|
|
logs
|
|
.*scalastyle-output.xml
|
|
.*dependency-reduced-pom.xml
|
|
known_translations
|
|
DESCRIPTION
|
|
NAMESPACE
|