[SPARK-16501][MESOS] Allow providing Mesos principal & secret via files
This commit modifies the Mesos submission client to allow the principal and secret to be provided indirectly via files. The path to these files can be specified either via Spark configuration or via environment variable. Assuming these files are appropriately protected by FS/OS permissions this means we don't ever leak the actual values in process info like ps Environment variable specification is useful because it allows you to interpolate the location of this file when using per-user Mesos credentials. For some background as to why we have taken this approach I will briefly describe our set up. On our systems we provide each authorised user account with their own Mesos credentials to provide certain security and audit guarantees to our customers. These credentials are managed by a central Secret management service. In our `spark-env.sh` we determine the appropriate secret and principal files to use depending on the user who is invoking Spark hence the need to inject these via environment variables as well as by configuration properties. So we set these environment variables appropriately and our Spark read in the contents of those files to authenticate itself with Mesos. This is functionality we have been using it in production across multiple customer sites for some time. This has been in the field for around 18 months with no reported issues. These changes have been sufficient to meet our customer security and audit requirements. We have been building and deploying custom builds of Apache Spark with various minor tweaks like this which we are now looking to contribute back into the community in order that we can rely upon stock Apache Spark builds and stop maintaining our own internal fork. Author: Rob Vesse <rvesse@dotnetrdf.org> Closes #20167 from rvesse/SPARK-16501.
This commit is contained in:
Rob Vesse
Marcelo Vanzin
parent
0fc26313f8
commit
7f10cf83f3
|
|
@ -82,6 +82,27 @@ a Spark driver program configured to connect to Mesos.
|
|||
Alternatively, you can also install Spark in the same location in all the Mesos slaves, and configure
|
||||
`spark.mesos.executor.home` (defaults to SPARK_HOME) to point to that location.
|
||||
|
||||
## Authenticating to Mesos
|
||||
|
||||
When Mesos Framework authentication is enabled it is necessary to provide a principal and secret by which to authenticate Spark to Mesos. Each Spark job will register with Mesos as a separate framework.
|
||||
|
||||
Depending on your deployment environment you may wish to create a single set of framework credentials that are shared across all users or create framework credentials for each user. Creating and managing framework credentials should be done following the Mesos [Authentication documentation](http://mesos.apache.org/documentation/latest/authentication/).
|
||||
|
||||
Framework credentials may be specified in a variety of ways depending on your deployment environment and security requirements. The most simple way is to specify the `spark.mesos.principal` and `spark.mesos.secret` values directly in your Spark configuration. Alternatively you may specify these values indirectly by instead specifying `spark.mesos.principal.file` and `spark.mesos.secret.file`, these settings point to files containing the principal and secret. These files must be plaintext files in UTF-8 encoding. Combined with appropriate file ownership and mode/ACLs this provides a more secure way to specify these credentials.
|
||||
|
||||
Additionally if you prefer to use environment variables you can specify all of the above via environment variables instead, the environment variable names are simply the configuration settings uppercased with `.` replaced with `_` e.g. `SPARK_MESOS_PRINCIPAL`.
|
||||
|
||||
### Credential Specification Preference Order
|
||||
|
||||
Please note that if you specify multiple ways to obtain the credentials then the following preference order applies. Spark will use the first valid value found and any subsequent values are ignored:
|
||||
|
||||
- `spark.mesos.principal` configuration setting
|
||||
- `SPARK_MESOS_PRINCIPAL` environment variable
|
||||
- `spark.mesos.principal.file` configuration setting
|
||||
- `SPARK_MESOS_PRINCIPAL_FILE` environment variable
|
||||
|
||||
An equivalent order applies for the secret. Essentially we prefer the configuration to be specified directly rather than indirectly by files, and we prefer that configuration settings are used over environment variables.
|
||||
|
||||
## Uploading Spark Package
|
||||
|
||||
When Mesos runs a task on a Mesos slave for the first time, that slave must have a Spark binary
|
||||
|
|
@ -427,7 +448,14 @@ See the [configuration page](configuration.html) for information on Spark config
|
|||
<td><code>spark.mesos.principal</code></td>
|
||||
<td>(none)</td>
|
||||
<td>
|
||||
Set the principal with which Spark framework will use to authenticate with Mesos.
|
||||
Set the principal with which Spark framework will use to authenticate with Mesos. You can also specify this via the environment variable `SPARK_MESOS_PRINCIPAL`.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>spark.mesos.principal.file</code></td>
|
||||
<td>(none)</td>
|
||||
<td>
|
||||
Set the file containing the principal with which Spark framework will use to authenticate with Mesos. Allows specifying the principal indirectly in more security conscious deployments. The file must be readable by the user launching the job and be UTF-8 encoded plaintext. You can also specify this via the environment variable `SPARK_MESOS_PRINCIPAL_FILE`.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
|
|
@ -435,7 +463,15 @@ See the [configuration page](configuration.html) for information on Spark config
|
|||
<td>(none)</td>
|
||||
<td>
|
||||
Set the secret with which Spark framework will use to authenticate with Mesos. Used, for example, when
|
||||
authenticating with the registry.
|
||||
authenticating with the registry. You can also specify this via the environment variable `SPARK_MESOS_SECRET`.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>spark.mesos.secret.file</code></td>
|
||||
<td>(none)</td>
|
||||
<td>
|
||||
Set the file containing the secret with which Spark framework will use to authenticate with Mesos. Used, for example, when
|
||||
authenticating with the registry. Allows for specifying the secret indirectly in more security conscious deployments. The file must be readable by the user launching the job and be UTF-8 encoded plaintext. You can also specify this via the environment variable `SPARK_MESOS_SECRET_FILE`.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@
|
|||
|
||||
package org.apache.spark.scheduler.cluster.mesos
|
||||
|
||||
import java.io.File
|
||||
import java.nio.charset.StandardCharsets
|
||||
import java.util.{List => JList}
|
||||
import java.util.concurrent.CountDownLatch
|
||||
|
||||
|
|
@ -25,6 +27,7 @@ import scala.collection.mutable.ArrayBuffer
|
|||
import scala.util.control.NonFatal
|
||||
|
||||
import com.google.common.base.Splitter
|
||||
import com.google.common.io.Files
|
||||
import org.apache.mesos.{MesosSchedulerDriver, Protos, Scheduler, SchedulerDriver}
|
||||
import org.apache.mesos.Protos.{TaskState => MesosTaskState, _}
|
||||
import org.apache.mesos.Protos.FrameworkInfo.Capability
|
||||
|
|
@ -71,26 +74,15 @@ trait MesosSchedulerUtils extends Logging {
|
|||
failoverTimeout: Option[Double] = None,
|
||||
frameworkId: Option[String] = None): SchedulerDriver = {
|
||||
val fwInfoBuilder = FrameworkInfo.newBuilder().setUser(sparkUser).setName(appName)
|
||||
val credBuilder = Credential.newBuilder()
|
||||
fwInfoBuilder.setHostname(Option(conf.getenv("SPARK_PUBLIC_DNS")).getOrElse(
|
||||
conf.get(DRIVER_HOST_ADDRESS)))
|
||||
webuiUrl.foreach { url => fwInfoBuilder.setWebuiUrl(url) }
|
||||
checkpoint.foreach { checkpoint => fwInfoBuilder.setCheckpoint(checkpoint) }
|
||||
failoverTimeout.foreach { timeout => fwInfoBuilder.setFailoverTimeout(timeout) }
|
||||
frameworkId.foreach { id =>
|
||||
fwInfoBuilder.setId(FrameworkID.newBuilder().setValue(id).build())
|
||||
}
|
||||
fwInfoBuilder.setHostname(Option(conf.getenv("SPARK_PUBLIC_DNS")).getOrElse(
|
||||
conf.get(DRIVER_HOST_ADDRESS)))
|
||||
conf.getOption("spark.mesos.principal").foreach { principal =>
|
||||
fwInfoBuilder.setPrincipal(principal)
|
||||
credBuilder.setPrincipal(principal)
|
||||
}
|
||||
conf.getOption("spark.mesos.secret").foreach { secret =>
|
||||
credBuilder.setSecret(secret)
|
||||
}
|
||||
if (credBuilder.hasSecret && !fwInfoBuilder.hasPrincipal) {
|
||||
throw new SparkException(
|
||||
"spark.mesos.principal must be configured when spark.mesos.secret is set")
|
||||
}
|
||||
|
||||
conf.getOption("spark.mesos.role").foreach { role =>
|
||||
fwInfoBuilder.setRole(role)
|
||||
}
|
||||
|
|
@ -98,6 +90,7 @@ trait MesosSchedulerUtils extends Logging {
|
|||
if (maxGpus > 0) {
|
||||
fwInfoBuilder.addCapabilities(Capability.newBuilder().setType(Capability.Type.GPU_RESOURCES))
|
||||
}
|
||||
val credBuilder = buildCredentials(conf, fwInfoBuilder)
|
||||
if (credBuilder.hasPrincipal) {
|
||||
new MesosSchedulerDriver(
|
||||
scheduler, fwInfoBuilder.build(), masterUrl, credBuilder.build())
|
||||
|
|
@ -106,6 +99,40 @@ trait MesosSchedulerUtils extends Logging {
|
|||
}
|
||||
}
|
||||
|
||||
def buildCredentials(
|
||||
conf: SparkConf,
|
||||
fwInfoBuilder: Protos.FrameworkInfo.Builder): Protos.Credential.Builder = {
|
||||
val credBuilder = Credential.newBuilder()
|
||||
conf.getOption("spark.mesos.principal")
|
||||
.orElse(Option(conf.getenv("SPARK_MESOS_PRINCIPAL")))
|
||||
.orElse(
|
||||
conf.getOption("spark.mesos.principal.file")
|
||||
.orElse(Option(conf.getenv("SPARK_MESOS_PRINCIPAL_FILE")))
|
||||
.map { principalFile =>
|
||||
Files.toString(new File(principalFile), StandardCharsets.UTF_8)
|
||||
}
|
||||
).foreach { principal =>
|
||||
fwInfoBuilder.setPrincipal(principal)
|
||||
credBuilder.setPrincipal(principal)
|
||||
}
|
||||
conf.getOption("spark.mesos.secret")
|
||||
.orElse(Option(conf.getenv("SPARK_MESOS_SECRET")))
|
||||
.orElse(
|
||||
conf.getOption("spark.mesos.secret.file")
|
||||
.orElse(Option(conf.getenv("SPARK_MESOS_SECRET_FILE")))
|
||||
.map { secretFile =>
|
||||
Files.toString(new File(secretFile), StandardCharsets.UTF_8)
|
||||
}
|
||||
).foreach { secret =>
|
||||
credBuilder.setSecret(secret)
|
||||
}
|
||||
if (credBuilder.hasSecret && !fwInfoBuilder.hasPrincipal) {
|
||||
throw new SparkException(
|
||||
"spark.mesos.principal must be configured when spark.mesos.secret is set")
|
||||
}
|
||||
credBuilder
|
||||
}
|
||||
|
||||
/**
|
||||
* Starts the MesosSchedulerDriver and stores the current running driver to this new instance.
|
||||
* This driver is expected to not be running.
|
||||
|
|
|
|||
|
|
@ -17,16 +17,20 @@
|
|||
|
||||
package org.apache.spark.scheduler.cluster.mesos
|
||||
|
||||
import java.io.{File, FileNotFoundException}
|
||||
|
||||
import scala.collection.JavaConverters._
|
||||
import scala.language.reflectiveCalls
|
||||
|
||||
import org.apache.mesos.Protos.{Resource, Value}
|
||||
import com.google.common.io.Files
|
||||
import org.apache.mesos.Protos.{FrameworkInfo, Resource, Value}
|
||||
import org.mockito.Mockito._
|
||||
import org.scalatest._
|
||||
import org.scalatest.mockito.MockitoSugar
|
||||
|
||||
import org.apache.spark.{SparkConf, SparkContext, SparkFunSuite}
|
||||
import org.apache.spark.{SparkConf, SparkContext, SparkException, SparkFunSuite}
|
||||
import org.apache.spark.internal.config._
|
||||
import org.apache.spark.util.SparkConfWithEnv
|
||||
|
||||
class MesosSchedulerUtilsSuite extends SparkFunSuite with Matchers with MockitoSugar {
|
||||
|
||||
|
|
@ -237,4 +241,157 @@ class MesosSchedulerUtilsSuite extends SparkFunSuite with Matchers with MockitoS
|
|||
val portsToUse = getRangesFromResources(resourcesToBeUsed).map{r => r._1}
|
||||
portsToUse.isEmpty shouldBe true
|
||||
}
|
||||
|
||||
test("Principal specified via spark.mesos.principal") {
|
||||
val conf = new SparkConf()
|
||||
conf.set("spark.mesos.principal", "test-principal")
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
}
|
||||
|
||||
test("Principal specified via spark.mesos.principal.file") {
|
||||
val pFile = File.createTempFile("MesosSchedulerUtilsSuite", ".txt");
|
||||
pFile.deleteOnExit()
|
||||
Files.write("test-principal".getBytes("UTF-8"), pFile);
|
||||
val conf = new SparkConf()
|
||||
conf.set("spark.mesos.principal.file", pFile.getAbsolutePath())
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
}
|
||||
|
||||
test("Principal specified via spark.mesos.principal.file that does not exist") {
|
||||
val conf = new SparkConf()
|
||||
conf.set("spark.mesos.principal.file", "/tmp/does-not-exist")
|
||||
|
||||
intercept[FileNotFoundException] {
|
||||
utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
}
|
||||
}
|
||||
|
||||
test("Principal specified via SPARK_MESOS_PRINCIPAL") {
|
||||
val conf = new SparkConfWithEnv(Map("SPARK_MESOS_PRINCIPAL" -> "test-principal"))
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
}
|
||||
|
||||
test("Principal specified via SPARK_MESOS_PRINCIPAL_FILE") {
|
||||
val pFile = File.createTempFile("MesosSchedulerUtilsSuite", ".txt");
|
||||
pFile.deleteOnExit()
|
||||
Files.write("test-principal".getBytes("UTF-8"), pFile);
|
||||
val conf = new SparkConfWithEnv(Map("SPARK_MESOS_PRINCIPAL_FILE" -> pFile.getAbsolutePath()))
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
}
|
||||
|
||||
test("Principal specified via SPARK_MESOS_PRINCIPAL_FILE that does not exist") {
|
||||
val conf = new SparkConfWithEnv(Map("SPARK_MESOS_PRINCIPAL_FILE" -> "/tmp/does-not-exist"))
|
||||
|
||||
intercept[FileNotFoundException] {
|
||||
utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
}
|
||||
}
|
||||
|
||||
test("Secret specified via spark.mesos.secret") {
|
||||
val conf = new SparkConf()
|
||||
conf.set("spark.mesos.principal", "test-principal")
|
||||
conf.set("spark.mesos.secret", "my-secret")
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
credBuilder.hasSecret shouldBe true
|
||||
credBuilder.getSecret shouldBe "my-secret"
|
||||
}
|
||||
|
||||
test("Principal specified via spark.mesos.secret.file") {
|
||||
val sFile = File.createTempFile("MesosSchedulerUtilsSuite", ".txt");
|
||||
sFile.deleteOnExit()
|
||||
Files.write("my-secret".getBytes("UTF-8"), sFile);
|
||||
val conf = new SparkConf()
|
||||
conf.set("spark.mesos.principal", "test-principal")
|
||||
conf.set("spark.mesos.secret.file", sFile.getAbsolutePath())
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
credBuilder.hasSecret shouldBe true
|
||||
credBuilder.getSecret shouldBe "my-secret"
|
||||
}
|
||||
|
||||
test("Principal specified via spark.mesos.secret.file that does not exist") {
|
||||
val conf = new SparkConf()
|
||||
conf.set("spark.mesos.principal", "test-principal")
|
||||
conf.set("spark.mesos.secret.file", "/tmp/does-not-exist")
|
||||
|
||||
intercept[FileNotFoundException] {
|
||||
utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
}
|
||||
}
|
||||
|
||||
test("Principal specified via SPARK_MESOS_SECRET") {
|
||||
val env = Map("SPARK_MESOS_SECRET" -> "my-secret")
|
||||
val conf = new SparkConfWithEnv(env)
|
||||
conf.set("spark.mesos.principal", "test-principal")
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
credBuilder.hasSecret shouldBe true
|
||||
credBuilder.getSecret shouldBe "my-secret"
|
||||
}
|
||||
|
||||
test("Principal specified via SPARK_MESOS_SECRET_FILE") {
|
||||
val sFile = File.createTempFile("MesosSchedulerUtilsSuite", ".txt");
|
||||
sFile.deleteOnExit()
|
||||
Files.write("my-secret".getBytes("UTF-8"), sFile);
|
||||
|
||||
val sFilePath = sFile.getAbsolutePath()
|
||||
val env = Map("SPARK_MESOS_SECRET_FILE" -> sFilePath)
|
||||
val conf = new SparkConfWithEnv(env)
|
||||
conf.set("spark.mesos.principal", "test-principal")
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
credBuilder.hasSecret shouldBe true
|
||||
credBuilder.getSecret shouldBe "my-secret"
|
||||
}
|
||||
|
||||
test("Secret specified with no principal") {
|
||||
val conf = new SparkConf()
|
||||
conf.set("spark.mesos.secret", "my-secret")
|
||||
|
||||
intercept[SparkException] {
|
||||
utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
}
|
||||
}
|
||||
|
||||
test("Principal specification preference") {
|
||||
val conf = new SparkConfWithEnv(Map("SPARK_MESOS_PRINCIPAL" -> "other-principal"))
|
||||
conf.set("spark.mesos.principal", "test-principal")
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
}
|
||||
|
||||
test("Secret specification preference") {
|
||||
val conf = new SparkConfWithEnv(Map("SPARK_MESOS_SECRET" -> "other-secret"))
|
||||
conf.set("spark.mesos.principal", "test-principal")
|
||||
conf.set("spark.mesos.secret", "my-secret")
|
||||
|
||||
val credBuilder = utils.buildCredentials(conf, FrameworkInfo.newBuilder())
|
||||
credBuilder.hasPrincipal shouldBe true
|
||||
credBuilder.getPrincipal shouldBe "test-principal"
|
||||
credBuilder.hasSecret shouldBe true
|
||||
credBuilder.getSecret shouldBe "my-secret"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue