spark-instrumented-optimizer/docs/security.md at 82a6a083a485140858bcd93d73adec59bb5cca64

Andrew Or 09f7e4587b [SPARK-2157] Enable tight firewall rules for Spark

The goal of this PR is to allow users of Spark to write tight firewall rules for their clusters. This is currently not possible because Spark uses random ports in many places, notably the communication between executors and drivers. The changes in this PR are based on top of ash211's changes in #1107.

The list covered here may or may not be the complete set of port needed for Spark to operate perfectly. However, as of the latest commit there are no known sources of random ports (except in tests). I have not documented a few of the more obscure configs.

My spark-env.sh looks like this:
```
export SPARK_MASTER_PORT=6060
export SPARK_WORKER_PORT=7070
export SPARK_MASTER_WEBUI_PORT=9090
export SPARK_WORKER_WEBUI_PORT=9091
```
and my spark-defaults.conf looks like this:
```
spark.master spark://andrews-mbp:6060
spark.driver.port 5001
spark.fileserver.port 5011
spark.broadcast.port 5021
spark.replClassServer.port 5031
spark.blockManager.port 5041
spark.executor.port 5051
```

Author: Andrew Or <andrewor14@gmail.com>
Author: Andrew Ash <andrew@andrewash.com>

Closes #1777 from andrewor14/configure-ports and squashes the following commits:

621267b [Andrew Or] Merge branch 'master' of github.com:apache/spark into configure-ports
8a6b820 [Andrew Or] Use a random UI port during tests
7da0493 [Andrew Or] Fix tests
523c30e [Andrew Or] Add test for isBindCollision
b97b02a [Andrew Or] Minor fixes
c22ad00 [Andrew Or] Merge branch 'master' of github.com:apache/spark into configure-ports
93d359f [Andrew Or] Executors connect to wrong port when collision occurs
d502e5f [Andrew Or] Handle port collisions when creating Akka systems
a2dd05c [Andrew Or] Patrick's comment nit
86461e2 [Andrew Or] Remove spark.executor.env.port and spark.standalone.client.port
1d2d5c6 [Andrew Or] Fix ports for standalone cluster mode
cb3be88 [Andrew Or] Various doc fixes (broken link, format etc.)
e837cde [Andrew Or] Remove outdated TODOs
bfbab28 [Andrew Or] Merge branch 'master' of github.com:apache/spark into configure-ports
de1b207 [Andrew Or] Update docs to reflect new ports
b565079 [Andrew Or] Add spark.ports.maxRetries
2551eb2 [Andrew Or] Remove spark.worker.watcher.port
151327a [Andrew Or] Merge branch 'master' of github.com:apache/spark into configure-ports
9868358 [Andrew Or] Add a few miscellaneous ports
6016e77 [Andrew Or] Add spark.executor.port
8d836e6 [Andrew Or] Also document SPARK_{MASTER/WORKER}_WEBUI_PORT
4d9e6f3 [Andrew Or] Fix super subtle bug
3f8e51b [Andrew Or] Correct erroneous docs...
e111d08 [Andrew Or] Add names for UI services
470f38c [Andrew Or] Special case non-"Address already in use" exceptions
1d7e408 [Andrew Or] Treat 0 ports specially + return correct ConnectionManager port
ba32280 [Andrew Or] Minor fixes
6b550b0 [Andrew Or] Assorted fixes
73fbe89 [Andrew Or] Move start service logic to Utils
ec676f4 [Andrew Or] Merge branch 'SPARK-2157' of github.com:ash211/spark into configure-ports
038a579 [Andrew Ash] Trust the server start function to report the port the service started on
7c5bdc4 [Andrew Ash] Fix style issue
0347aef [Andrew Ash] Unify port fallback logic to a single place
24a4c32 [Andrew Ash] Remove type on val to match surrounding style
9e4ad96 [Andrew Ash] Reformat for style checker
5d84e0e [Andrew Ash] Document new port configuration options
066dc7a [Andrew Ash] Fix up HttpServer port increments
cad16da [Andrew Ash] Add fallover increment logic for HttpServer
c5a0568 [Andrew Ash] Fix ConnectionManager to retry with increment
b80d2fd [Andrew Ash] Make Spark's block manager port configurable
17c79bb [Andrew Ash] Add a configuration option for spark-shell's class server
f34115d [Andrew Ash] SPARK-1176 Add port configuration for HttpBroadcast
49ee29b [Andrew Ash] SPARK-1174 Add port configuration for HttpFileServer
1c0981a [Andrew Ash] Make port in HttpServer configurable

2014-08-06 00:07:40 -07:00

7 KiB

Raw Blame History

layout	title
global	Spark Security

Spark currently supports authentication via a shared secret. Authentication can be configured to be on via the spark.authenticate configuration parameter. This parameter controls whether the Spark communication protocols do authentication using the shared secret. This authentication is a basic handshake to make sure both sides have the same shared secret and are allowed to communicate. If the shared secret is not identical they will not be allowed to communicate. The shared secret is created as follows:

For Spark on YARN deployments, configuring spark.authenticate to true will automatically handle generating and distributing the shared secret. Each application will use a unique shared secret.
For other types of Spark deployments, the Spark parameter spark.authenticate.secret should be configured on each of the nodes. This secret will be used by all the Master/Workers and applications.
IMPORTANT NOTE: The experimental Netty shuffle path (spark.shuffle.use.netty) is not secured, so do not use Netty for shuffles if running with authentication.

Web UI

The Spark UI can also be secured by using javax servlet filters via the spark.ui.filters setting. A user may want to secure the UI if it has data that other users should not be allowed to see. The javax servlet filter specified by the user can authenticate the user and then once the user is logged in, Spark can compare that user versus the view ACLs to make sure they are authorized to view the UI. The configs spark.acls.enable and spark.ui.view.acls control the behavior of the ACLs. Note that the user who started the application always has view access to the UI. On YARN, the Spark UI uses the standard YARN web application proxy mechanism and will authenticate via any installed Hadoop filters.

Spark also supports modify ACLs to control who has access to modify a running Spark application. This includes things like killing the application or a task. This is controlled by the configs spark.acls.enable and spark.modify.acls. Note that if you are authenticating the web UI, in order to use the kill button on the web UI it might be necessary to add the users in the modify acls to the view acls also. On YARN, the modify acls are passed in and control who has modify access via YARN interfaces.

Spark allows for a set of administrators to be specified in the acls who always have view and modify permissions to all the applications. is controlled by the config spark.admin.acls. This is useful on a shared cluster where you might have administrators or support staff who help users debug applications.

Event Logging

If your applications are using event logging, the directory where the event logs go (spark.eventLog.dir) should be manually created and have the proper permissions set on it. If you want those log files secured, the permissions should be set to drwxrwxrwxt for that directory. The owner of the directory should be the super user who is running the history server and the group permissions should be restricted to super user group. This will allow all users to write to the directory but will prevent unprivileged users from removing or renaming a file unless they own the file or directory. The event log files will be created by Spark with permissions such that only the user and group have read and write access.

Configuring Ports for Network Security

Spark makes heavy use of the network, and some environments have strict requirements for using tight firewall settings. Below are the primary ports that Spark uses for its communication and how to configure those ports.

Standalone mode only

From	To	Default Port	Purpose	Configuration Setting	Notes
Browser	Standalone Master	8080	Web UI	`spark.master.ui.port / SPARK_MASTER_WEBUI_PORT`	Jetty-based. Standalone mode only.
Browser	Standalone Worker	8081	Web UI	`spark.worker.ui.port / SPARK_WORKER_WEBUI_PORT`	Jetty-based. Standalone mode only.
Driver / Standalone Worker	Standalone Master	7077	Submit job to cluster / Join cluster	`SPARK_MASTER_PORT`	Akka-based. Set to "0" to choose a port randomly. Standalone mode only.
Standalone Master	Standalone Worker	(random)	Schedule executors	`SPARK_WORKER_PORT`	Akka-based. Set to "0" to choose a port randomly. Standalone mode only.

All cluster managers

From	To	Default Port	Purpose	Configuration Setting	Notes
Browser	Application	4040	Web UI	`spark.ui.port`	Jetty-based
Browser	History Server	18080	Web UI	`spark.history.ui.port`	Jetty-based
Executor / Standalone Master	Driver	(random)	Connect to application / Notify executor state changes	`spark.driver.port`	Akka-based. Set to "0" to choose a port randomly.
Driver	Executor	(random)	Schedule tasks	`spark.executor.port`	Akka-based. Set to "0" to choose a port randomly.
Executor	Driver	(random)	File server for files and jars	`spark.fileserver.port`	Jetty-based
Executor	Driver	(random)	HTTP Broadcast	`spark.broadcast.port`	Jetty-based. Not used by TorrentBroadcast, which sends data through the block manager instead.
Executor	Driver	(random)	Class file server	`spark.replClassServer.port`	Jetty-based. Only used in Spark shells.
Executor / Driver	Executor / Driver	(random)	Block Manager port	`spark.blockManager.port`	Raw socket via ServerSocketChannel

See the configuration page for more details on the security configuration parameters, and org.apache.spark.SecurityManager for implementation details about security.

7 KiB Raw Blame History