ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/ec2
History
Nicholas Chammas 50820f1527 [SPARK-1805] [EC2] Validate instance types 
Addresses [SPARK-1805](https://issues.apache.org/jira/browse/SPARK-1805), though doesn't resolve it completely.

Error out quickly if the user asks for the master and slaves to have different AMI virtualization types, since we don't currently support that.

In addition to that, we print warnings if the inputted instance types are not recognized, though I would prefer if we errored out. Elsewhere in the script it seems [we allow unrecognized instance types](5de14cc276/ec2/spark_ec2.py (L331)), though I think we should remove that.

It's messy, but it should serve us until we enhance spark-ec2 to support clusters with mixed virtualization types.

Author: Nicholas Chammas <nicholas.chammas@gmail.com>

Closes #4455 from nchammas/ec2-master-slave-different-virtualization and squashes the following commits:

ce28609 [Nicholas Chammas] fix style
b0adba0 [Nicholas Chammas] validate input instance types
2015-02-10 15:45:38 +00:00
..
deploy.generic/root/spark-ec2 [SPARK-787] Add S3 configuration parameters to the EC2 deploy scripts 2014-09-16 13:40:16 -07:00
README Remove remaining references to incubation 2014-03-02 01:00:16 -08:00
spark-ec2 [SPARK-5434] [EC2] Preserve spaces in EC2 path 2015-01-28 12:56:03 -08:00
spark_ec2.py [SPARK-1805] [EC2] Validate instance types 2015-02-10 15:45:38 +00:00

README 

This folder contains a script, spark-ec2, for launching Spark clusters on
Amazon EC2. Usage instructions are available online at:

http://spark.apache.org/docs/latest/ec2-scripts.html