[SPARK-30263][CORE] Don't log potentially sensitive value of non-Spark properties ignored in spark-submit

### What changes were proposed in this pull request?

The value of non-Spark config properties ignored in spark-submit is no longer logged.

### Why are the changes needed?

The value isn't really needed in the logs, and could contain potentially sensitive info. While we can redact the values selectively too, I figured it's more robust to just not log them at all here, as the values aren't important in this log statement.

### Does this PR introduce any user-facing change?

Other than the change to logging above, no.

### How was this patch tested?

Existing tests

Closes #26893 from srowen/SPARK-30263.

Authored-by: Sean Owen <srowen@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
Sean Owen 2019-12-14 13:13:54 -08:00 committed by Dongjoon Hyun
parent d3ec8b1735
commit 46e950bea8

View file

@ -137,10 +137,10 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
* Remove keys that don't start with "spark." from `sparkProperties`.
*/
private def ignoreNonSparkProperties(): Unit = {
sparkProperties.foreach { case (k, v) =>
sparkProperties.keys.foreach { k =>
if (!k.startsWith("spark.")) {
sparkProperties -= k
logWarning(s"Ignoring non-spark config property: $k=$v")
logWarning(s"Ignoring non-Spark config property: $k")
}
}
}