ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-29466][WEBUI] Show Duration for running drivers in Standalone master web UI

Browse source 
### What changes were proposed in this pull request?

This PR aims to add a new column `Duration` for running drivers in Apache Spark `Standalone` master web UI in order to improve UX. This help users like the other `Duration` columns in the `Running` and `Completed` application tables.

### Why are the changes needed?

When we use `--supervise`, the drivers can survive longer.
Technically, the `Duration` column is not the same. (Please see the image below.)

### Does this PR introduce any user-facing change?

Yes. The red box is added newly.

<img width="1312" alt="Screen Shot 2019-10-14 at 12 53 43 PM" src="https://user-images.githubusercontent.com/9700541/66779127-50301b80-ee82-11e9-853f-72222cd011ac.png">

### How was this patch tested?

Manual since this is a UI column. After starting standalone cluster and jobs, kill the `DriverWrapper` and see the UI.

```
$ sbin/start-master.sh
$ sbin/start-slave.sh spark://$(hostname):7077
$ bin/spark-submit --master spark://(hostname):7077 --deploy-mode cluster --supervise --class org.apache.spark.examples.JavaSparkPi examples/target/scala-2.12/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar 1000
$ jps
41521 DriverWrapper
...
$ kill -9 41521   // kill the `DriverWrapper`.
```

Closes #26113 from dongjoon-hyun/SPARK-29466.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
Dongjoon Hyun 2019-10-18 15:39:44 -07:00
parent 4cfce3e5d0
commit e4b4a35de2
1 changed files with 17 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

22
core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala
View file

@ -108,12 +108,17 @@ private[ui] class MasterPage(parent: MasterWebUI) extends WebUIPage("") {
val completedApps = state.completedApps.sortBy(_.endTime).reverse
val completedAppsTable = UIUtils.listingTable(appHeaders, appRow, completedApps)
val driverHeaders = Seq("Submission ID", "Submitted Time", "Worker", "State", "Cores",
"Memory", "Resources", "Main Class")
val activeDriverHeaders = Seq("Submission ID", "Submitted Time", "Worker", "State", "Cores",
"Memory", "Resources", "Main Class", "Duration")
val activeDrivers = state.activeDrivers.sortBy(_.startTime).reverse
val activeDriversTable = UIUtils.listingTable(driverHeaders, driverRow, activeDrivers)
val activeDriversTable =
UIUtils.listingTable(activeDriverHeaders, activeDriverRow, activeDrivers)
val completedDriverHeaders = Seq("Submission ID", "Submitted Time", "Worker", "State", "Cores",
"Memory", "Resources", "Main Class")
val completedDrivers = state.completedDrivers.sortBy(_.startTime).reverse
val completedDriversTable = UIUtils.listingTable(driverHeaders, driverRow, completedDrivers)
val completedDriversTable =
UIUtils.listingTable(completedDriverHeaders, completedDriverRow, completedDrivers)
// For now we only show driver information if the user has submitted drivers to the cluster.
// This is until we integrate the notion of drivers and applications in the UI.
@ -310,7 +315,11 @@ private[ui] class MasterPage(parent: MasterWebUI) extends WebUIPage("") {
</tr>
}
private def driverRow(driver: DriverInfo): Seq[Node] = {
private def activeDriverRow(driver: DriverInfo) = driverRow(driver, showDuration = true)
private def completedDriverRow(driver: DriverInfo) = driverRow(driver, showDuration = false)
private def driverRow(driver: DriverInfo, showDuration: Boolean): Seq[Node] = {
val killLink = if (parent.killEnabled &&
(driver.state == DriverState.RUNNING ||
driver.state == DriverState.SUBMITTED ||
@ -345,6 +354,9 @@ private[ui] class MasterPage(parent: MasterWebUI) extends WebUIPage("") {
</td>
<td>{formatResourcesAddresses(driver.resources)}</td>
<td>{driver.desc.command.arguments(2)}</td>
{if (showDuration) {
<td>{UIUtils.formatDuration(System.currentTimeMillis() - driver.startTime)}</td>
}}
</tr>
}
}