EC2 script and job scheduling documentation still refered to Shark.
I removed these references.
I also removed a remaining `SHARK_VERSION` variable from `ec2-variables.sh`.
Author: Pierre Borckmans <pierre.borckmans@realimpactanalytics.com>
Closes#5083 from pierre-borckmans/remove_refererences_to_shark_in_docs and squashes the following commits:
4e90ffc [Pierre Borckmans] Removed deprecated SHARK_VERSION
caea407 [Pierre Borckmans] Remove shark reference from ec2 script doc
196c744 [Pierre Borckmans] Removed references to Shark
This PR comes from Tachyon community to solve the issue:
https://tachyon.atlassian.net/browse/TACHYON-11
An accompanying PR is in mesos/spark-ec2:
https://github.com/mesos/spark-ec2/pull/101
Author: cheng chang <myairia@gmail.com>
Closes#4901 from uronce-cc/master and squashes the following commits:
313aa36 [cheng chang] minor re-wording
fd2a48e [cheng chang] Remove Tachyon when deploying through git hash
1d53c5c [cheng chang] add default value to --tachyon-version
6f8887e [cheng chang] make tachyon version configurable
When deploying to AWS, there is additional configuration that is required to read S3 files. EMR creates it automatically, there is no reason that the Spark EC2 script shouldn't.
This PR requires a corresponding PR to the mesos/spark-ec2 to be merged, as it gets cloned in the process of setting up machines: https://github.com/mesos/spark-ec2/pull/58
Author: Dan Osipov <daniil.osipov@shazam.com>
Closes#1120 from danosipov/s3_credentials and squashes the following commits:
758da8b [Dan Osipov] Modify documentation to include the new parameter
71fab14 [Dan Osipov] Use a parameter --copy-aws-credentials to enable S3 credential deployment
7e0da26 [Dan Osipov] Get AWS credentials out of boto connection instance
39bdf30 [Dan Osipov] Add S3 configuration parameters to the EC2 deploy scripts
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS
Depends on: https://github.com/mesos/spark-ec2/pull/46
Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>
Closes#612 from douglaz/ec2_configurable_workers and squashes the following commits:
d6c5d65 [Allan Douglas R. de Oliveira] Added master opts parameter
6c34671 [Allan Douglas R. de Oliveira] Use number of worker instances as string on template
ba528b9 [Allan Douglas R. de Oliveira] Added SPARK_WORKER_INSTANCES parameter
