[SPARK-30263][CORE] Don't log potentially sensitive value of non-Spark properties ignored in spark-submit
### What changes were proposed in this pull request? The value of non-Spark config properties ignored in spark-submit is no longer logged. ### Why are the changes needed? The value isn't really needed in the logs, and could contain potentially sensitive info. While we can redact the values selectively too, I figured it's more robust to just not log them at all here, as the values aren't important in this log statement. ### Does this PR introduce any user-facing change? Other than the change to logging above, no. ### How was this patch tested? Existing tests Closes #26893 from srowen/SPARK-30263. Authored-by: Sean Owen <srowen@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
parent
d3ec8b1735
commit
46e950bea8
|
@ -137,10 +137,10 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
|
|||
* Remove keys that don't start with "spark." from `sparkProperties`.
|
||||
*/
|
||||
private def ignoreNonSparkProperties(): Unit = {
|
||||
sparkProperties.foreach { case (k, v) =>
|
||||
sparkProperties.keys.foreach { k =>
|
||||
if (!k.startsWith("spark.")) {
|
||||
sparkProperties -= k
|
||||
logWarning(s"Ignoring non-spark config property: $k=$v")
|
||||
logWarning(s"Ignoring non-Spark config property: $k")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
Loading…
Reference in a new issue