ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-30371][K8S] Add spark.kubernetes.driver.master conf

Browse source 
### What changes were proposed in this pull request?

make KUBERNETES_MASTER_INTERNAL_URL configurable

### Why are the changes needed?

we do not always use the default port number 443 to access our kube-apiserver, and even in some mulit-tenant cluster,  people do not use the service `kubernetes.default.svc` to access the kube-apiserver, so make the internal master configurable is necessary。

### Does this PR introduce any user-facing change?

user can configure the internal master url by
```
--conf spark.kubernetes.internal.master=https://kubernetes.default.svc:6443
```

### How was this patch tested?

run in multi-cluster that do not use the https://kubernetes.default.svc to access the kube-apiserver

Closes #27029 from wackxu/internalmaster.

Authored-by: xushiwei 00425595 <xushiwei5@huawei.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
xushiwei 00425595 2020-01-19 14:14:45 -08:00 committed by Dongjoon Hyun
parent d4c6ec6ba7
commit f14061c6a4
3 changed files with 15 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
docs/running-on-kubernetes.md
View file

@ -507,6 +507,13 @@ See the [configuration page](configuration.html) for information on Spark config
configuration properties e.g. <code>spark.kubernetes.namespace</code>.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.driver.master</code></td>
<td><code>https://kubernetes.default.svc</code></td>
<td>
The internal Kubernetes master (API server) address to be used for driver to request executors.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.namespace</code></td>
<td><code>default</code></td>

7
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
View file

@ -36,6 +36,13 @@ private[spark] object Config extends Logging {
.stringConf
.createOptional
val KUBERNETES_DRIVER_MASTER_URL =
ConfigBuilder("spark.kubernetes.driver.master")
.doc("The internal Kubernetes master (API server) address " +
"to be used for driver to request executors.")
.stringConf
.createWithDefault(KUBERNETES_MASTER_INTERNAL_URL)
val KUBERNETES_NAMESPACE =
ConfigBuilder("spark.kubernetes.namespace")
.doc("The namespace that will be used for running the driver and executor pods.")

2
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
View file

@ -51,7 +51,7 @@ private[spark] class KubernetesClusterManager extends ExternalClusterManager wit
"If the application is deployed using spark-submit in cluster mode, the driver pod name " +
"must be provided.")
(KUBERNETES_AUTH_DRIVER_MOUNTED_CONF_PREFIX,
KUBERNETES_MASTER_INTERNAL_URL,
sc.conf.get(KUBERNETES_DRIVER_MASTER_URL),
Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)),
Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)))
} else {