ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/docs/img
History
Luca Canali fd308ade52 [SPARK-30041][SQL][WEBUI] Add Codegen Stage Id to Stage DAG visualization in Web UI 
### What changes were proposed in this pull request?
SPARK-29894 provides information on the Codegen Stage Id in WEBUI for SQL Plan graphs. Similarly, this proposes to add Codegen Stage Id in the DAG visualization for Stage execution. DAGs for Stage execution are available in the WEBUI under the Jobs and Stages tabs.

### Why are the changes needed?
This is proposed as an aid for drill-down analysis of complex SQL statement execution, as it is not always easy to match parts of the SQL Plan graph with the corresponding Stage DAG execution graph. Adding Codegen Stage Id for WholeStageCodegen operations makes this task easier.

### Does this PR introduce any user-facing change?
Stage DAG visualization in the WEBUI will show codegen stage id for WholeStageCodegen operations, as in the example snippet from the WEBUI, Jobs tab  (the query used in the example is TPCDS 2.4 q14a):
![](https://issues.apache.org/jira/secure/attachment/12987461/Snippet_StagesDags_with_CodegenId%20_annotated.png)

### How was this patch tested?
Manually tested, see also example snippet.

Closes #26675 from LucaCanali/addCodegenStageIdtoStageGraph.

Authored-by: Luca Canali <luca.canali@cern.ch>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
2020-01-18 01:00:45 +08:00
..
AllJobsPageDetail1.png [SPARK-28543][DOCS][WEBUI] Document Spark Jobs page 2019-08-15 08:52:23 -05:00
AllJobsPageDetail2.png [SPARK-28543][DOCS][WEBUI] Document Spark Jobs page 2019-08-15 08:52:23 -05:00
AllJobsPageDetail3.png [SPARK-28543][DOCS][WEBUI] Document Spark Jobs page 2019-08-15 08:52:23 -05:00
AllStagesPageDetail1.png [SPARK-28542][DOCS][WEBUI] Stages Tab 2019-08-31 13:33:44 -05:00
AllStagesPageDetail2.png [SPARK-28542][DOCS][WEBUI] Stages Tab 2019-08-31 13:33:44 -05:00
AllStagesPageDetail3.png [SPARK-28542][DOCS][WEBUI] Stages Tab 2019-08-31 13:33:44 -05:00
AllStagesPageDetail4.png [SPARK-30041][SQL][WEBUI] Add Codegen Stage Id to Stage DAG visualization in Web UI 2020-01-18 01:00:45 +08:00
AllStagesPageDetail5.png [SPARK-30041][SQL][WEBUI] Add Codegen Stage Id to Stage DAG visualization in Web UI 2020-01-18 01:00:45 +08:00
AllStagesPageDetail6.png [SPARK-28542][DOCS][WEBUI] Stages Tab 2019-08-31 13:33:44 -05:00
AllStagesPageDetail7.png [SPARK-28542][DOCS][WEBUI] Stages Tab 2019-08-31 13:33:44 -05:00
AllStagesPageDetail8.png [SPARK-28542][DOCS][WEBUI] Stages Tab 2019-08-31 13:33:44 -05:00
AllStagesPageDetail9.png [SPARK-28542][DOCS][WEBUI] Stages Tab 2019-08-31 13:33:44 -05:00
cluster-overview.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
cluster-overview.pptx [SPARK-6343] Doc driver-worker network reqs 2015-04-09 06:37:20 -04:00
edge_cut_vs_vertex_cut.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
glyphicons-halflings-white.png
glyphicons-halflings.png
graphx_figures.pptx
graphx_logo.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
java-sm.png
JDBCServer1.png [SPARK-28373][DOCS][WEBUI] JDBC/ODBC Server Tab 2019-09-14 10:18:52 -07:00
JDBCServer2.png [SPARK-28373][DOCS][WEBUI] JDBC/ODBC Server Tab 2019-09-14 10:18:52 -07:00
JDBCServer3.png [SPARK-28373][DOCS][WEBUI] JDBC/ODBC Server Tab 2019-09-14 10:18:52 -07:00
JobPageDetail1.png [SPARK-28543][DOCS][WEBUI] Document Spark Jobs page 2019-08-15 08:52:23 -05:00
JobPageDetail2.png [SPARK-30041][SQL][WEBUI] Add Codegen Stage Id to Stage DAG visualization in Web UI 2020-01-18 01:00:45 +08:00
JobPageDetail3.png [SPARK-28543][DOCS][WEBUI] Document Spark Jobs page 2019-08-15 08:52:23 -05:00
k8s-cluster-mode.png [SPARK-22648][K8S] Spark on Kubernetes - Documentation 2017-12-21 17:21:11 -08:00
ml-Pipeline.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
ml-PipelineModel.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
ml-Pipelines.pptx [SPARK-4575] [mllib] [docs] spark.ml pipelines doc + bug fixes 2014-12-04 17:00:06 +08:00
property_graph.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
python-sm.png
scala-sm.png
spark-logo-hd.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
spark-webui-accumulators.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
streaming-arch.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
streaming-dstream-ops.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
streaming-dstream-window.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
streaming-dstream.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
streaming-figures.pptx [SPARK-2419][Streaming][Docs] More updates to the streaming programming guide 2014-09-06 14:46:43 -07:00
streaming-flow.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
streaming-kinesis-arch.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
structured-streaming-example-model.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
structured-streaming-late-data.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
structured-streaming-model.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
structured-streaming-stream-as-a-table.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
structured-streaming-watermark-append-mode.png [SPARK-19074][SS][DOCS] Updated Structured Streaming Programming Guide for update mode and source/sink options 2017-01-06 11:29:01 -08:00
structured-streaming-watermark-update-mode.png [SPARK-19074][SS][DOCS] Updated Structured Streaming Programming Guide for update mode and source/sink options 2017-01-06 11:29:01 -08:00
structured-streaming-window.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
structured-streaming.pptx [SPARK-19074][SS][DOCS] Updated Structured Streaming Programming Guide for update mode and source/sink options 2017-01-06 11:29:01 -08:00
triplet.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
vertex_routing_edge_tables.png [MINOR][DOCS] Remove unused images; crush PNGs that could use it for good measure 2016-07-04 09:21:58 +01:00
webui-env-class.png [SPARK-28540][WEBUI] Document Environment page 2019-08-21 10:48:48 -05:00
webui-env-hadoop.png [SPARK-28540][WEBUI] Document Environment page 2019-08-21 10:48:48 -05:00
webui-env-sys.png [SPARK-28540][WEBUI] Document Environment page 2019-08-21 10:48:48 -05:00
webui-env-tab.png [SPARK-28540][WEBUI] Document Environment page 2019-08-21 10:48:48 -05:00
webui-exe-err.png [SPARK-28539][WEBUI][DOC] Document Executors page 2019-08-28 08:34:24 -05:00
webui-exe-tab.png [SPARK-28539][WEBUI][DOC] Document Executors page 2019-08-28 08:34:24 -05:00
webui-exe-thread.png [SPARK-28539][WEBUI][DOC] Document Executors page 2019-08-28 08:34:24 -05:00
webui-sql-dag.png [SPARK-29894][SQL][WEBUI] Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab 2019-11-20 23:20:33 +08:00
webui-sql-plan.png [SPARK-28538][UI] Document SQL page 2019-08-12 08:36:01 -05:00
webui-sql-tab.png [SPARK-28539][WEBUI][DOC] Document Executors page 2019-08-28 08:34:24 -05:00
webui-storage-detail.png [SPARK-28541][WEBUI] Document Storage page 2019-08-20 20:05:13 -05:00
webui-storage-tab.png [SPARK-28541][WEBUI] Document Storage page 2019-08-20 20:05:13 -05:00