ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-31394][DOC][FOLLOWUP] Add nfs volume type description

Browse source 
### What changes were proposed in this pull request?

This adds newly supported `nfs` volume type description into the document for Apache Spark 3.1.0.

### Why are the changes needed?

To complete the document.

### Does this PR introduce any user-facing change?

Yes. (Doc)

![nfs_screen_shot](https://user-images.githubusercontent.com/9700541/79530887-8f077f80-8025-11ea-8cc1-e0b551802d5d.png)

### How was this patch tested?

Manually generate doc and check it.
```
SKIP_API=1 jekyll build
```

Closes #28236 from dongjoon-hyun/SPARK-NFS-DOC.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This commit is contained in:
Dongjoon Hyun 2020-04-17 12:07:34 -07:00
parent 61b7d446b3
commit fde996be87
No known key found for this signature in database
GPG key ID: EDA00CE834F0FC5C
1 changed files with 12 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
docs/running-on-kubernetes.md
View file

@ -274,10 +274,11 @@ will be the driver or executor container.
## Using Kubernetes Volumes
Starting with Spark 2.4.0, users can mount the following types of Kubernetes [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) into the driver and executor pods:
Users can mount the following types of Kubernetes [volumes](https://kubernetes.io/docs/concepts/storage/volumes/) into the driver and executor pods:
* [hostPath](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath): mounts a file or directory from the host node’s filesystem into a pod.
* [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir): an initially empty volume created when a pod is assigned to a node.
* [persistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim): used to mount a `PersistentVolume` into a pod.
* [nfs](https://kubernetes.io/docs/concepts/storage/volumes/#nfs): mounts an existing NFS(Network File System) into a pod.
* [persistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim): mounts a `PersistentVolume` into a pod.
**NB:** Please see the [Security](#security) section of this document for security issues related to volume mounts.
@ -289,7 +290,7 @@ To mount a volume of any of the types above into the driver pod, use the followi
--conf spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.subPath=<mount subPath>
```
Specifically, `VolumeType` can be one of the following values: `hostPath`, `emptyDir`, and `persistentVolumeClaim`. `VolumeName` is the name you want to use for the volume under the `volumes` field in the pod specification.
Specifically, `VolumeType` can be one of the following values: `hostPath`, `emptyDir`, `nfs` and `persistentVolumeClaim`. `VolumeName` is the name you want to use for the volume under the `volumes` field in the pod specification.
Each supported type of volumes may have some specific configuration options, which can be specified using configuration properties of the following form:
@ -297,7 +298,14 @@ Each supported type of volumes may have some specific configuration options, whi
spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].options.[OptionName]=<value>
```
For example, the claim name of a `persistentVolumeClaim` with volume name `checkpointpvc` can be specified using the following property:
For example, the server and path of a `nfs` with volume name `images` can be specified using the following properties:
```
spark.kubernetes.driver.volumes.nfs.images.options.server=example.com
spark.kubernetes.driver.volumes.nfs.images.options.path=/data
```
And, the claim name of a `persistentVolumeClaim` with volume name `checkpointpvc` can be specified using the following property:
```
spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.claimName=check-point-pvc-claim