ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-27244][CORE] Redact Passwords While Using Option logConf=true

Browse source 
## What changes were proposed in this pull request?

When logConf is set to true, config keys that contain password were printed in cleartext in driver log. This change uses the already present redact method in Utils, to redact all the passwords based on redact pattern in SparkConf and then print the conf to driver log thus ensuring that sensitive information like passwords is not printed in clear text.

## How was this patch tested?
This patch was tested through `SparkConfSuite` & then entire unit test through sbt

Please review http://spark.apache.org/contributing.html before opening a pull request.

Closes #24196 from ninadingole/SPARK-27244.

Authored-by: Ninad Ingole <robert.wallis@example.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
This commit is contained in:
Ninad Ingole 2019-03-29 14:16:53 -05:00 committed by Sean Owen
parent 06abd06112
commit dbc7ce18b9
2 changed files with 10 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
core/src/main/scala/org/apache/spark/SparkConf.scala
View file

@ -606,7 +606,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging with Seria
* configuration out for debugging.
*/
def toDebugString: String = {
getAll.sorted.map{case (k, v) => k + "=" + v}.mkString("\n")
Utils.redact(this, getAll).sorted.map { case (k, v) => k + "=" + v }.mkString("\n")
}
}

10
core/src/test/scala/org/apache/spark/SparkConfSuite.scala
View file

@ -32,7 +32,7 @@ import org.apache.spark.internal.config.Kryo._
import org.apache.spark.internal.config.Network._
import org.apache.spark.network.util.ByteUnit
import org.apache.spark.serializer.{JavaSerializer, KryoRegistrator, KryoSerializer}
import org.apache.spark.util.{ResetSystemProperties, RpcUtils}
import org.apache.spark.util.{ResetSystemProperties, RpcUtils, Utils}
class SparkConfSuite extends SparkFunSuite with LocalSparkContext with ResetSystemProperties {
test("Test byteString conversion") {
@ -354,6 +354,14 @@ class SparkConfSuite extends SparkFunSuite with LocalSparkContext with ResetSyst
}
}
test("SPARK-27244 toDebugString should redact passwords") {
val conf = new SparkConf().set("dummy.password", "dummy-password")
conf.validateSettings()
assert(conf.get("dummy.password") === "dummy-password")
assert(conf.toDebugString.contains(s"dummy.password=${Utils.REDACTION_REPLACEMENT_TEXT}"))
}
val defaultIllegalValue = "SomeIllegalValue"
val illegalValueTests : Map[String, (SparkConf, String) => Any] = Map(
"getTimeAsSeconds" -> (_.getTimeAsSeconds(_)),