ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-4913] Fix incorrect event log path

Browse source 
SPARK-2261 uses a single file to log events for an app. `eventLogDir` in `ApplicationDescription` is replaced with `eventLogFile`. However, `ApplicationDescription` in `SparkDeploySchedulerBackend` is initialized with `SparkContext`'s `eventLogDir`. It is just the log directory, not the actual log file path. `Master.rebuildSparkUI` can not correctly rebuild a new SparkUI for the app.

Because the `ApplicationDescription` is remotely registered with `Master` and the app's id is then generated in `Master`, we can not get the app id in advance before registration. So the received description needs to be modified with correct `eventLogFile` value.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes #3755 from viirya/fix_app_logdir and squashes the following commits:

5e0ea35 [Liang-Chi Hsieh] Revision for comment.
b5730a1 [Liang-Chi Hsieh] Fix incorrect event log path.

Closes #3777 (a duplicate PR for the same JIRA)
This commit is contained in:
Liang-Chi Hsieh 2014-12-23 14:58:33 -08:00 committed by Josh Rosen
parent 27c5399f4d
commit 96281cd0c3
2 changed files with 17 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala
View file

@ -23,7 +23,7 @@ private[spark] class ApplicationDescription(
val memoryPerSlave: Int,
val command: Command,
var appUiUrl: String,
val eventLogFile: Option[String] = None)
val eventLogDir: Option[String] = None)
extends Serializable {
val user = System.getProperty("user.name", "<unknown>")

20
core/src/main/scala/org/apache/spark/deploy/master/Master.scala
View file

@ -719,14 +719,26 @@ private[spark] class Master(
def rebuildSparkUI(app: ApplicationInfo): Boolean = {
val appName = app.desc.name
val notFoundBasePath = HistoryServer.UI_PATH_PREFIX + "/not-found"
val eventLogFile = app.desc.eventLogFile.getOrElse {
// Event logging is not enabled for this application
app.desc.appUiUrl = notFoundBasePath
val eventLogFile = app.desc.eventLogDir
.map { dir => EventLoggingListener.getLogPath(dir, app.id) }
.getOrElse {
// Event logging is not enabled for this application
app.desc.appUiUrl = notFoundBasePath
return false
}
val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf)
if (fs.exists(new Path(eventLogFile + EventLoggingListener.IN_PROGRESS))) {
// Event logging is enabled for this application, but the application is still in progress
val title = s"Application history not found (${app.id})"
var msg = s"Application $appName is still in progress."
logWarning(msg)
msg = URLEncoder.encode(msg, "UTF-8")
app.desc.appUiUrl = notFoundBasePath + s"?msg=$msg&title=$title"
return false
}
try {
val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf)
val (logInput, sparkVersion) = EventLoggingListener.openEventLog(new Path(eventLogFile), fs)
val replayBus = new ReplayListenerBus()
val ui = SparkUI.createHistoryUI(new SparkConf, replayBus, new SecurityManager(conf),