8ed93c9355
### What changes were proposed in this pull request? This PR fixes a wrong coloring issue in the DAG-viz. In the Job Page and Stage Page, nodes which are associated with "barrier mode" in the DAG-viz will be colored pale green. But, with some type of jobs, nodes which are not associated with the mode will also colored. You can reproduce with the following operation. ``` sc.parallelize(1 to 10).barrier.mapPartitions(identity).repartition(1).collect() ``` <img width="376" alt="wrong-coloring" src="https://user-images.githubusercontent.com/4736016/83403670-1711df00-a444-11ea-9457-c683f75bc566.png"> In the screen shot above, `repartition` in `Stage 1` is not associated with barrier mode so the corresponding node should not be colored pale green. The cause of this issue is the logic which chooses HTML elements to be colored is wrong. The logic chooses such elements based on whether each element is associated with a style class (`clusterId` in the code). But when an operation crosses over shuffle (like `repartition` above), a `clusterId` can be duplicated and non-barrier mode node is also associated with the same `clusterId`. ### Why are the changes needed? This is a bug. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Newly added test case with the following command. ``` build/sbt -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite -- -z SPARK-31886" ``` Closes #28694 from sarutak/fix-wrong-barrier-color. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Gengliang Wang <gengliang.wang@databricks.com> |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| src | ||
| pom.xml | ||
