stop, start and destroy require the EC2_REGION
i.e ./spark-ec2 --region=us-west-1 stop yourclustername Author: Jeff Steinmetz <jeffrey.steinmetz@gmail.com> Closes #2473 from jeffsteinmetz/master and squashes the following commits: 7491f2c [Jeff Steinmetz] fix case in EC2 cluster setup documentation bd3d777 [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args 2bf4a57 [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args 68d8372 [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args d2ab6e2 [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args 520e6dc [Jeff Steinmetz] standardized ec2 documenation to use <lower-case> sample args 37fc876 [Jeff Steinmetz] stop, start and destroy require the EC2_REGION
This commit is contained in:
parent
d8a9d1d442
commit
9e8ced7847
|
@ -48,6 +48,15 @@ by looking for the "Name" tag of the instance in the Amazon EC2 Console.
|
||||||
key pair, `<num-slaves>` is the number of slave nodes to launch (try
|
key pair, `<num-slaves>` is the number of slave nodes to launch (try
|
||||||
1 at first), and `<cluster-name>` is the name to give to your
|
1 at first), and `<cluster-name>` is the name to give to your
|
||||||
cluster.
|
cluster.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export AWS_SECRET_ACCESS_KEY=AaBbCcDdEeFGgHhIiJjKkLlMmNnOoPpQqRrSsTtU
|
||||||
|
export AWS_ACCESS_KEY_ID=ABCDEFG1234567890123
|
||||||
|
./spark-ec2 --key-pair=awskey --identity-file=awskey.pem --region=us-west-1 --zone=us-west-1a --spark-version=1.1.0 launch my-spark-cluster
|
||||||
|
```
|
||||||
|
|
||||||
- After everything launches, check that the cluster scheduler is up and sees
|
- After everything launches, check that the cluster scheduler is up and sees
|
||||||
all the slaves by going to its web UI, which will be printed at the end of
|
all the slaves by going to its web UI, which will be printed at the end of
|
||||||
the script (typically `http://<master-hostname>:8080`).
|
the script (typically `http://<master-hostname>:8080`).
|
||||||
|
@ -55,27 +64,27 @@ by looking for the "Name" tag of the instance in the Amazon EC2 Console.
|
||||||
You can also run `./spark-ec2 --help` to see more usage options. The
|
You can also run `./spark-ec2 --help` to see more usage options. The
|
||||||
following options are worth pointing out:
|
following options are worth pointing out:
|
||||||
|
|
||||||
- `--instance-type=<INSTANCE_TYPE>` can be used to specify an EC2
|
- `--instance-type=<instance-type>` can be used to specify an EC2
|
||||||
instance type to use. For now, the script only supports 64-bit instance
|
instance type to use. For now, the script only supports 64-bit instance
|
||||||
types, and the default type is `m1.large` (which has 2 cores and 7.5 GB
|
types, and the default type is `m1.large` (which has 2 cores and 7.5 GB
|
||||||
RAM). Refer to the Amazon pages about [EC2 instance
|
RAM). Refer to the Amazon pages about [EC2 instance
|
||||||
types](http://aws.amazon.com/ec2/instance-types) and [EC2
|
types](http://aws.amazon.com/ec2/instance-types) and [EC2
|
||||||
pricing](http://aws.amazon.com/ec2/#pricing) for information about other
|
pricing](http://aws.amazon.com/ec2/#pricing) for information about other
|
||||||
instance types.
|
instance types.
|
||||||
- `--region=<EC2_REGION>` specifies an EC2 region in which to launch
|
- `--region=<ec2-region>` specifies an EC2 region in which to launch
|
||||||
instances. The default region is `us-east-1`.
|
instances. The default region is `us-east-1`.
|
||||||
- `--zone=<EC2_ZONE>` can be used to specify an EC2 availability zone
|
- `--zone=<ec2-zone>` can be used to specify an EC2 availability zone
|
||||||
to launch instances in. Sometimes, you will get an error because there
|
to launch instances in. Sometimes, you will get an error because there
|
||||||
is not enough capacity in one zone, and you should try to launch in
|
is not enough capacity in one zone, and you should try to launch in
|
||||||
another.
|
another.
|
||||||
- `--ebs-vol-size=GB` will attach an EBS volume with a given amount
|
- `--ebs-vol-size=<GB>` will attach an EBS volume with a given amount
|
||||||
of space to each node so that you can have a persistent HDFS cluster
|
of space to each node so that you can have a persistent HDFS cluster
|
||||||
on your nodes across cluster restarts (see below).
|
on your nodes across cluster restarts (see below).
|
||||||
- `--spot-price=PRICE` will launch the worker nodes as
|
- `--spot-price=<price>` will launch the worker nodes as
|
||||||
[Spot Instances](http://aws.amazon.com/ec2/spot-instances/),
|
[Spot Instances](http://aws.amazon.com/ec2/spot-instances/),
|
||||||
bidding for the given maximum price (in dollars).
|
bidding for the given maximum price (in dollars).
|
||||||
- `--spark-version=VERSION` will pre-load the cluster with the
|
- `--spark-version=<version>` will pre-load the cluster with the
|
||||||
specified version of Spark. VERSION can be a version number
|
specified version of Spark. The `<version>` can be a version number
|
||||||
(e.g. "0.7.3") or a specific git hash. By default, a recent
|
(e.g. "0.7.3") or a specific git hash. By default, a recent
|
||||||
version will be used.
|
version will be used.
|
||||||
- If one of your launches fails due to e.g. not having the right
|
- If one of your launches fails due to e.g. not having the right
|
||||||
|
@ -137,11 +146,11 @@ cost you any EC2 cycles, but ***will*** continue to cost money for EBS
|
||||||
storage.
|
storage.
|
||||||
|
|
||||||
- To stop one of your clusters, go into the `ec2` directory and run
|
- To stop one of your clusters, go into the `ec2` directory and run
|
||||||
`./spark-ec2 stop <cluster-name>`.
|
`./spark-ec2 --region=<ec2-region> stop <cluster-name>`.
|
||||||
- To restart it later, run
|
- To restart it later, run
|
||||||
`./spark-ec2 -i <key-file> start <cluster-name>`.
|
`./spark-ec2 -i <key-file> --region=<ec2-region> start <cluster-name>`.
|
||||||
- To ultimately destroy the cluster and stop consuming EBS space, run
|
- To ultimately destroy the cluster and stop consuming EBS space, run
|
||||||
`./spark-ec2 destroy <cluster-name>` as described in the previous
|
`./spark-ec2 --region=<ec2-region> destroy <cluster-name>` as described in the previous
|
||||||
section.
|
section.
|
||||||
|
|
||||||
# Limitations
|
# Limitations
|
||||||
|
|
Loading…
Reference in a new issue