Commit graph

116 commits

Author SHA1 Message Date
Basit Mustafa 7f87ab9813 Added t2 instance types
New t2 instance types require HVM amis, bailout assumption of pvm
causes failures when using t2 instance types.

Author: Basit Mustafa <basitmustafa@computes-things-for-basit.local>

Closes #1446 from 24601/master and squashes the following commits:

01fe128 [Basit Mustafa] Makin' it pretty
392a95e [Basit Mustafa] Added t2 instance types
2014-07-18 12:23:47 -07:00
Nicholas Chammas 369aa84e8f name ec2 instances and security groups consistently
Security groups created by `spark-ec2` do not prepend “spark-“ to the
name.

Since naming the instances themselves is new to `spark-ec2`, it’s better
to change that pattern to match the existing naming pattern for the
security groups, rather than the other way around.

Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Author: nchammas <nicholas.chammas@gmail.com>

Closes #1344 from nchammas/master and squashes the following commits:

f7e4581 [Nicholas Chammas] unrelated pep8 fix
a36eed0 [Nicholas Chammas] name ec2 instances and security groups consistently
de7292a [nchammas] Merge pull request #4 from apache/master
2e4fe00 [nchammas] Merge pull request #3 from apache/master
89fde08 [nchammas] Merge pull request #2 from apache/master
69f6e22 [Nicholas Chammas] PEP8 fixes
2627247 [Nicholas Chammas] broke up lines before they hit 100 chars
6544b7e [Nicholas Chammas] [SPARK-2065] give launched instances names
69da6cf [nchammas] Merge pull request #1 from apache/master
2014-07-10 12:56:00 -07:00
Andrew Or 56e009d4f0 [EC2] Add default history server port to ec2 script
Right now I have to open it manually

Author: Andrew Or <andrewor14@gmail.com>

Closes #1296 from andrewor14/hist-serv-port and squashes the following commits:

8895a1f [Andrew Or] Add default history server port to ec2 script
2014-07-08 16:49:31 +09:00
Zichuan Ye 62d4a0fa99 Fixing AWS instance type information based upon current EC2 data
Fixed a problem in previous file in which some information regarding AWS instance types were wrong. Such information was updated base upon current AWS EC2 data.

Author: Zichuan Ye <jerry@tangentds.com>

Closes #1156 from jerry86/master and squashes the following commits:

ff36e95 [Zichuan Ye] Fixing AWS instance type information based upon current EC2 data
2014-06-26 15:21:29 -07:00
Jean-Martin Archer 9cb64b2c54 SPARK-2166 - Listing of instances to be terminated before the prompt
Will list the EC2 instances before detroying the cluster.
This was added because it can be scary to destroy EC2
instances without knowing which one will be impacted.

Author: Jean-Martin Archer <jeanmartin.archer@pulseenergy.com>

This patch had conflicts when merged, resolved by
Committer: Patrick Wendell <pwendell@gmail.com>

Closes #270 from j-martin/master and squashes the following commits:

826455f [Jean-Martin Archer] [SPARK-2611] Implementing recommendations
27b0a36 [Jean-Martin Archer] Listing of instances to be terminated before the prompt Will list the EC2 instances before detroying the cluster. This was added because it can be scary to destroy EC2 instances without knowing which one will be impacted.
2014-06-22 20:54:42 -07:00
Ori Kremer 9fc373e3a9 SPARK-2241: quote command line args in ec2 script
To preserve quoted command line args (in case options have space in them).

Author: Ori Kremer <ori.kremer@gmail.com>

Closes #1169 from orikremer/quote_cmd_line_args and squashes the following commits:

67e2aa1 [Ori Kremer] quote command line args
2014-06-22 20:23:49 -07:00
Patrick Wendell b2ebf429e2 HOTFIX: bug caused by #941
This patch should have qualified the use of PIPE. This needs to be back ported into 0.9 and 1.0.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #1108 from pwendell/hotfix and squashes the following commits:

711c58d [Patrick Wendell] HOTFIX: bug caused by #941
2014-06-17 15:09:24 -07:00
Anant 8cd04c3eec SPARK-1990: added compatibility for python 2.6 for ssh_read command
https://issues.apache.org/jira/browse/SPARK-1990

There were some posts on the lists that spark-ec2 does not work with Python 2.6. In addition, we should check the Python version at the top of the script and exit if it's too old

Author: Anant <anant.asty@gmail.com>

Closes #941 from anantasty/SPARK-1990 and squashes the following commits:

4ca441d [Anant] Implmented check_optput withinthe module to work with python 2.6
c6ed85c [Anant] added compatibility for python 2.6 for ssh_read command
2014-06-16 23:43:01 -07:00
Nicholas Chammas a2052a44f3 [SPARK-2065] give launched instances names
This update resolves [SPARK-2065](https://issues.apache.org/jira/browse/SPARK-2065). It gives launched EC2 instances descriptive names by using instance tags. Launched instances now show up in the EC2 console with these names.

I used `format()` with named parameters, which I believe is the recommended practice for string formatting in Python, but which doesn’t seem to be used elsewhere in the script.

Author: Nicholas Chammas <nicholas.chammas@gmail.com>
Author: nchammas <nicholas.chammas@gmail.com>

Closes #1043 from nchammas/master and squashes the following commits:

69f6e22 [Nicholas Chammas] PEP8 fixes
2627247 [Nicholas Chammas] broke up lines before they hit 100 chars
6544b7e [Nicholas Chammas] [SPARK-2065] give launched instances names
69da6cf [nchammas] Merge pull request #1 from apache/master
2014-06-10 21:49:08 -07:00
Varakhedi Sujeet 11ded3f66f SPARK-1790: Update EC2 scripts to support r3 instance types
Author: Varakhedi Sujeet <svarakhedi@gopivotal.com>

Closes #960 from sujeetv/ec2-r3 and squashes the following commits:

3cb9fd5 [Varakhedi Sujeet] SPARK-1790: Update EC2 scripts to support r3 instance
2014-06-04 16:02:23 -07:00
Aaron Davidson ab7c62d573 Update spark-ec2 scripts for 1.0.0 on master
The change was previously committed only to branch-1.0 as part of a34e6fda1d

Author: Aaron Davidson <aaron@databricks.com>

This patch had conflicts when merged, resolved by
Committer: Patrick Wendell <pwendell@gmail.com>

Closes #938 from aarondav/sparkec2 and squashes the following commits:

067cc31 [Aaron Davidson] Update spark-ec2 scripts for 1.0.0 on master
2014-06-03 22:33:04 -07:00
Reynold Xin eea3aab4f2 Made spark_ec2.py PEP8 compliant.
The change set is actually pretty small -- mostly whitespace changes. Admittedly this is a scary change due to the lack of tests to cover the ec2 scripts, and also because indentation actually impacts control flow in Python ...

Look at changes without whitespace diff here: https://github.com/apache/spark/pull/891/files?w=1

Author: Reynold Xin <rxin@apache.org>

Closes #891 from rxin/spark-ec2-pep8 and squashes the following commits:

ac1bf11 [Reynold Xin] Made spark_ec2.py PEP8 compliant.
2014-06-01 15:39:04 -07:00
Patrick Wendell c0ab85d732 Version bump of spark-ec2 scripts
This will allow us to change things in spark-ec2 related to the 1.0 release.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #809 from pwendell/spark-ec2 and squashes the following commits:

59117fb [Patrick Wendell] Version bump of spark-ec2 scripts
2014-05-16 21:42:14 -07:00
msiddalingaiah bb2bb0cf6e Address SPARK-1717
I tested the change locally with Spark 0.9.1, but I can't test with 1.0.0 because there was no AMI for it at the time. It's a trivial fix, so it shouldn't cause any problems.

Author: msiddalingaiah <madhu@madhu.com>

Closes #641 from msiddalingaiah/master and squashes the following commits:

a4f7404 [msiddalingaiah] Address SPARK-1717
2014-05-04 21:59:10 -07:00
Allan Douglas R. de Oliveira bcb9b7fd4a EC2 script should exit with non-zero code on UsageError
This is specially import because some ssh errors are raised as UsageError, preventing an automated usage of the script from detecting the failure.

Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>

Closes #638 from douglaz/ec2_exit_code_fix and squashes the following commits:

5915e6d [Allan Douglas R. de Oliveira] EC2 script should exit with non-zero code on UsageError
2014-05-04 20:36:51 -07:00
Allan Douglas R. de Oliveira 4669a84ab1 EC2 configurable workers
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS

Depends on: https://github.com/mesos/spark-ec2/pull/46

Author: Allan Douglas R. de Oliveira <allan@chaordicsystems.com>

Closes #612 from douglaz/ec2_configurable_workers and squashes the following commits:

d6c5d65 [Allan Douglas R. de Oliveira] Added master opts parameter
6c34671 [Allan Douglas R. de Oliveira] Use number of worker instances as string on template
ba528b9 [Allan Douglas R. de Oliveira] Added SPARK_WORKER_INSTANCES parameter
2014-05-03 16:52:19 -07:00
Harvey Feng 7b4203ab4c Add Spark v0.9.1 to ec2 launch script and use it as the default
Mainly ported from branch-0.9.

Author: Harvey Feng <hyfeng224@gmail.com>

Closes #385 from harveyfeng/0.9.1-ec2 and squashes the following commits:

769ac2f [Harvey Feng] Add Spark v0.9.1 to ec2 launch script and use it as the default
2014-04-10 18:25:54 -07:00
CodingCat 3eb009f362 SPARK-1156: allow user to login into a cluster without slaves
Reported in https://spark-project.atlassian.net/browse/SPARK-1156

The current spark-ec2 script doesn't allow user to login to a cluster without slaves. One of the issues brought by this behaviour is that when all the worker died, the user cannot even login to the cluster for debugging, etc.

Author: CodingCat <zhunansjtu@gmail.com>

Closes #58 from CodingCat/SPARK-1156 and squashes the following commits:

104af07 [CodingCat] output ERROR to stderr
9a71769 [CodingCat] do not allow user to start 0-slave cluster
24a7c79 [CodingCat] allow user to login into a cluster without slaves
2014-03-05 21:47:34 -08:00
Patrick Wendell 1fd2bfd3dd Remove remaining references to incubation
This removes some loose ends not caught by the other (incubating -> tlp) patches. @markhamstra this updates the version as you mentioned earlier.

Author: Patrick Wendell <pwendell@gmail.com>

Closes #51 from pwendell/tlp and squashes the following commits:

d553b1b [Patrick Wendell] Remove remaining references to incubation
2014-03-02 01:00:16 -08:00
Xiangrui Meng b61435c7ff SPARK-1106: check key name and identity file before launch a cluster
I launched an EC2 cluster without providing a key name and an identity file. The error showed up after two minutes. It would be good to check those options before launch, given the fact that EC2 billing rounds up to hours.

JIRA: https://spark-project.atlassian.net/browse/SPARK-1106

Author: Xiangrui Meng <meng@databricks.com>

Closes #617 from mengxr/ec2 and squashes the following commits:

2dfb316 [Xiangrui Meng] check key name and identity file before launch a cluster
2014-02-18 18:30:02 -08:00
Shivaram Venkataraman 2414ed310e Merge pull request #598 from shivaram/master.
Update spark_ec2 to use 0.9.0 by default

Backports change from branch-0.9

Author: Shivaram Venkataraman <shivaram@eecs.berkeley.edu>

Closes #598 and squashes the following commits:

f6d3ed0 [Shivaram Venkataraman] Update spark_ec2 to use 0.9.0 by default Backports change from branch-0.9
2014-02-13 14:26:06 -08:00
Christian Lundgren 5fa53c02fc Add c3 instance types to Spark EC2
The number of disks for the c3 instance types taken from here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#StorageOnInstanceTypes

Author: Christian Lundgren <christian.lundgren@gameanalytics.com>

Closes #595 from chrisavl/branch-0.9 and squashes the following commits:

c8af5f9 [Christian Lundgren] Add c3 instance types to Spark EC2
(cherry picked from commit 19b4bb2b44)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
2014-02-13 12:46:47 -08:00
Shivaram Venkataraman 7c4e6e1bf1 Add i2 instance types to Spark EC2. 2014-01-10 12:44:55 -08:00
Prashant Sharma 59e8009b8d a few left over document change 2014-01-02 21:48:44 +05:30
Ewen Cheslack-Postava d17c142615 Force pseudo-tty allocation in spark-ec2 script.
ssh commands need the -t argument repeated twice if there is no local
tty, e.g. if the process running spark-ec2 uses nohup and the parent
process exits.
2013-12-16 08:09:37 -08:00
Ankur Dave bc9f7eacb9 Enable stopping and starting a spot cluster 2013-11-11 17:50:31 -08:00
Haoyuan Li 6f455553c9 expose UI port only 2013-11-10 16:00:09 -08:00
Haoyuan Li 77cedf81c7 add tachyon module 2013-11-09 22:52:23 -08:00
Fabrizio (Misto) Milo 4b5d61f31f update default github 2013-11-01 18:41:49 -07:00
Patrick Wendell 3745a1827f Adding Shark 0.7.1 to EC2 scripts 2013-10-07 15:03:42 -07:00
Patrick Wendell 2484b84678 Bumping EC2 default version in master to 0.8.0. 2013-10-05 16:59:11 -07:00
CruncherBigData c85f720588 Update README 2013-10-01 09:05:03 -07:00
Reynold Xin 76677b8fa1 Merge pull request #670 from jey/ec2-ssh-improvements
EC2 SSH improvements
2013-09-26 14:03:46 -07:00
Aaron Davidson f589ce771a Fix issue with spark_ec2 seeing empty security groups
Under unknown, but occasional, circumstances, reservation.groups is empty
despite reservation.instances each having groups. This means that the
spark_ec2 get_existing_clusters() method would fail to find any instances.
To fix it, we simply use the instances' groups as the source of truth.

Note that this is actually just a revival of PR #827, now that the issue
has been reproduced.
2013-09-19 14:09:26 -07:00
Jey Kottalam e86d1d4a52 Clarify error messages on SSH failure 2013-09-11 14:59:42 -07:00
Patrick Wendell bddf135670 Change port from 3030 to 4040 2013-09-11 10:01:38 -07:00
Patrick Wendell 6d2198643c Adding Apache license to two files 2013-09-07 20:46:58 -07:00
Jey Kottalam b98572c70a Generate new SSH key for the cluster, make "--identity-file" optional 2013-09-06 14:51:47 -07:00
Jey Kottalam 6919a28d51 Construct shell commands as sequences for safety and composability 2013-09-06 14:28:26 -07:00
Patrick Wendell c02585ea13 Make initial connection failure message less daunting.
Right now it seems like something has gone wrong when this message is printed out.
Instead, this is a normal condition. So I changed the message a bit.
2013-08-21 15:45:45 -07:00
Patrick Wendell 6be6b71c8c Merge branch 'master' into ec2-updates
Conflicts:
	ec2/spark_ec2.py
2013-08-21 15:34:31 -07:00
Patrick Wendell 51a1a0c602 Bump spark version 2013-08-20 22:14:52 -07:00
Patrick Wendell 07e5c8b695 Set default Hadoop version to 1 2013-08-20 15:49:52 -07:00
Matei Zaharia 793a722f8e Allow some wiggle room in UISuite port test and in EC2 ports 2013-08-19 18:51:00 -07:00
Matei Zaharia 498a26189b Small fixes to web UI:
- Use SPARK_PUBLIC_DNS environment variable if set (for EC2)
- Use a non-ephemeral port (3030 instead of 33000) by default
- Updated test to use non-ephemeral port too
2013-08-19 18:17:49 -07:00
Patrick Wendell 7e1b7d8a96 Adding web UI port 2013-08-08 14:17:11 -07:00
Patrick Wendell 1105766295 Point to new repository/branch 2013-08-01 15:43:41 -07:00
Patrick Wendell 56715587e4 Pointing to new EC2 repository 2013-08-01 12:27:43 -07:00
Patrick Wendell 5cc725a0e3 Merge branch 'master' into ec2-updates
Conflicts:
	ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh
2013-07-31 21:35:12 -07:00
Patrick Wendell 893aaff738 Don't install MapReduce for Hadoop1 2013-07-30 14:55:04 -07:00