Jey Kottalam
6919a28d51
Construct shell commands as sequences for safety and composability
2013-09-06 14:28:26 -07:00
Patrick Wendell
c02585ea13
Make initial connection failure message less daunting.
...
Right now it seems like something has gone wrong when this message is printed out.
Instead, this is a normal condition. So I changed the message a bit.
2013-08-21 15:45:45 -07:00
Patrick Wendell
6be6b71c8c
Merge branch 'master' into ec2-updates
...
Conflicts:
ec2/spark_ec2.py
2013-08-21 15:34:31 -07:00
Patrick Wendell
51a1a0c602
Bump spark version
2013-08-20 22:14:52 -07:00
Patrick Wendell
07e5c8b695
Set default Hadoop version to 1
2013-08-20 15:49:52 -07:00
Matei Zaharia
793a722f8e
Allow some wiggle room in UISuite port test and in EC2 ports
2013-08-19 18:51:00 -07:00
Matei Zaharia
498a26189b
Small fixes to web UI:
...
- Use SPARK_PUBLIC_DNS environment variable if set (for EC2)
- Use a non-ephemeral port (3030 instead of 33000) by default
- Updated test to use non-ephemeral port too
2013-08-19 18:17:49 -07:00
Patrick Wendell
7e1b7d8a96
Adding web UI port
2013-08-08 14:17:11 -07:00
Patrick Wendell
1105766295
Point to new repository/branch
2013-08-01 15:43:41 -07:00
Patrick Wendell
56715587e4
Pointing to new EC2 repository
2013-08-01 12:27:43 -07:00
Patrick Wendell
5cc725a0e3
Merge branch 'master' into ec2-updates
...
Conflicts:
ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh
2013-07-31 21:35:12 -07:00
Patrick Wendell
893aaff738
Don't install MapReduce for Hadoop1
2013-07-30 14:55:04 -07:00
Matei Zaharia
207548b67b
Open up Job UI ports (33000-33010) on EC2 clusters
2013-07-29 17:19:33 -07:00
Ubuntu
88a0823c58
Consistently invoke bash with /usr/bin/env bash in scripts to make code more portable (JIRA Ticket SPARK-817)
2013-07-18 00:51:18 +00:00
Patrick Wendell
31b3c1df54
Small improvement in python script
2013-07-17 17:43:15 -07:00
Matei Zaharia
af3c9d5042
Add Apache license headers and LICENSE and NOTICE files
2013-07-16 17:21:33 -07:00
Patrick Wendell
8dccee16af
Bug fix
2013-07-11 16:50:27 -07:00
Patrick Wendell
31c18a2528
Removing support for custom Shark version
2013-07-11 16:18:16 -07:00
Patrick Wendell
13809d363c
Small fix
2013-07-11 15:45:35 -07:00
Patrick Wendell
f1689185d6
Adding ability to custom deploy only Spark
2013-07-11 15:44:31 -07:00
Patrick Wendell
311e63f2c7
Removing mesos terminology
2013-07-11 14:13:10 -07:00
Patrick Wendell
13e3dd98d8
Removing mesos support
2013-06-26 11:36:39 -07:00
Patrick Wendell
d5f74aa9b7
Cleanup of AMI tag
2013-06-13 10:30:59 -07:00
Patrick Wendell
eae7590d0d
Support for multiple hadoop versions
2013-06-12 18:39:01 -07:00
Patrick Wendell
bb328c8ff8
Re-working spark versions to use templates
2013-06-11 20:14:33 -07:00
Patrick Wendell
2124563a00
Remving support for old scripts
2013-06-11 17:52:10 -07:00
Patrick Wendell
8637c02191
Minor doc cleanup
2013-06-07 14:54:26 -07:00
Patrick Wendell
0f4c0e461f
Removing now defunct ami documentation
2013-06-07 14:39:58 -07:00
Patrick Wendell
d65d3dd8f0
Fixing syntax error
2013-06-07 14:16:01 -07:00
Patrick Wendell
2477736237
Adding spark shark modules
2013-06-07 11:55:46 -07:00
Patrick Wendell
db50379737
Decoupling spark version from ami
2013-06-07 11:54:16 -07:00
Patrick Wendell
a07bd286bb
Some changes in response to Josh
2013-05-18 16:00:32 -07:00
Patrick Wendell
e8a1d02956
Adding support for tag
2013-05-10 15:13:22 -07:00
Patrick Wendell
2197b2159a
Removing unnecessary parsing
2013-05-09 11:27:13 -07:00
Patrick Wendell
36c117a1f1
Slight change to AMI versioning
2013-05-08 22:50:53 -07:00
Patrick Wendell
f6c965a98b
Changing spark version and availability zone fix
2013-05-08 21:46:01 -07:00
Patrick Wendell
976e02491e
Resolve AMI region automatically
2013-05-08 21:46:01 -07:00
Patrick Wendell
f8213aab4b
Adding automatic resolve of AMI
2013-05-08 21:46:01 -07:00
Patrick Wendell
03b58378d0
Adding mapreduce module
2013-05-08 21:46:01 -07:00
Patrick Wendell
73f8cd237e
Adding extra instance types
2013-05-08 21:46:01 -07:00
Patrick Wendell
9ff8ad4bec
REVERT ME: Linking to Patrick's ec2 repo for now
2013-05-08 21:46:00 -07:00
Josh Rosen
e014c1d1cb
Fix SPARK-670: EC2 start command should require -i option.
2013-05-05 11:30:36 -07:00
Holden Karau
9bf24e1d61
Just use a loop for retries
2013-04-09 21:37:02 -07:00
Holden Karau
ff2130a0ad
Retry failed ssh commands. This is especially useful during system startup when the hosts may not have yet come on-line but can be useful at other points for people with flakey connections
2013-04-06 00:35:50 -07:00
Matei Zaharia
baa30fcd99
Use new Spark EC2 scripts by default
2013-02-26 23:38:50 -08:00
Shivaram Venkataraman
6cba5a48b0
Print cluster url after setup completes
2013-02-18 18:30:36 -08:00
Shivaram Venkataraman
e7cdf7a6a4
Print ganglia url after setup
2013-02-18 17:15:22 -08:00
Shivaram Venkataraman
03f45a18d5
Use port 5080 for httpd/ganglia
2013-02-18 16:56:01 -08:00
Matei Zaharia
455d015076
Clean up EC2 script options a bit
2013-02-17 16:53:12 -08:00
Matei Zaharia
08e444df0e
Change EC2 script to use 0.6 AMIs by default, for now
2013-02-17 14:01:48 -08:00
Shivaram Venkataraman
bf675ab4f9
Turn on ganglia by default
2013-01-31 21:43:45 -08:00
Shivaram Venkataraman
da44a391fb
Add an option to use the old scripts
2013-01-28 11:16:14 -08:00
Shivaram Venkataraman
dc9d3ab6ed
Add option to start ganglia. Also enable Hadoop ports even if cluster type is
...
not mesos
2013-01-27 00:26:00 -08:00
Shivaram Venkataraman
0243b081ce
Fix swap variable name
2013-01-27 00:25:29 -08:00
Shivaram Venkataraman
d012cfa0e8
Update spark_ec2.py to use new spark-ec2 scripts
2013-01-26 22:48:39 -08:00
Matei Zaharia
391e5a194a
Merge pull request #323 from admobius/master
...
SPARK-626: Remove security groups properly
2012-12-11 12:01:42 -08:00
Peter Sankauskas
f97ce3ae14
SPARK-626: Making security group deletion optional, handling retried
...
when deleting security groups fails, fixing bug when using all zones but
only 1 slave.
2012-12-11 10:48:21 -08:00
Peter Sankauskas
9f964612a1
SPARK-626: Remove rules before removing security groups, with a pause in
...
between so wait for AWS eventual consistency to catch up.
2012-12-10 17:44:09 -08:00
Matei Zaharia
597520ae20
Make sure the SSH key we copy to EC2 has permissions 600.
...
SPARK-539 #resolve
2012-12-10 15:12:06 -08:00
Matei Zaharia
cd16eab0db
Merge pull request #309 from admobius/use-boto-config-file
...
Improved use of Boto
2012-11-19 15:51:41 -08:00
Peter Sankauskas
dc2fb3c4b6
Allow Boto to use the other config options it supports, and gracefully
...
handling Boto connection exceptions (like AuthFailure)
2012-11-19 14:21:16 -08:00
Matei Zaharia
85ce5f27c1
Merge pull request #308 from admobius/multi-zone
...
Let EC2 script launch slaves in multiple availability zones
2012-11-19 13:24:09 -08:00
Peter Sankauskas
606d252d26
Adding comment about additional bandwidth charges
2012-11-17 23:09:11 -08:00
Peter Sankauskas
32442ee1e1
Giving the Spark EC2 script the ability to launch instances spread
...
across multiple availability zones in order to make the cluster more
resilient to failure
2012-11-16 17:25:28 -08:00
Peter Sankauskas
6d22f7ccb8
Delete security groups when deleting the cluster. As many operations
...
are done on instances in specific security groups, this seems like a
reasonable thing to clean up.
2012-11-16 14:02:43 -08:00
Josh Rosen
594eed31c4
Fix check for existing instances during EC2 launch.
2012-11-03 17:02:47 -07:00
Josh Rosen
96c9bcfd8d
Cancel spot instance requests when exiting spark-ec2.
2012-10-30 23:32:38 -07:00
Josh Rosen
365a4c1e68
Allow EC2 script to stop/destroy cluster after master/slave failures.
2012-10-18 10:36:50 -07:00
Matei Zaharia
5b7ee173e1
Update EC2 scripts for Spark 0.6
2012-10-12 19:53:03 -07:00
Ray Racine
c52bd1b6b1
Add m1.medium node option to cluster management script.
2012-10-09 11:05:13 -04:00
Matei Zaharia
47507d69d9
Made region used by spark-ec2 configurable.
2012-08-28 22:40:00 -07:00
Denny
8dc7242544
Use root login in standalone AMI
2012-08-14 10:18:24 -07:00
Denny
7152c7c12d
rsync root directory in EC2 script
2012-08-14 09:26:47 -07:00
Matei Zaharia
8069bd5b41
Removed separate launcher for EC2 standalone cluster
2012-08-09 22:45:24 +02:00
Denny
aaed039e36
Merged standalone and mesos EC2 scripts
2012-08-02 15:23:52 -07:00
Denny
0ee44c225e
Spark standalone mode cluster scripts.
...
Heavily inspired by Hadoop cluster scripts ;-)
2012-08-01 20:38:52 -07:00
Matei Zaharia
8a5bd4e765
Increase the default wait time for EC2 clusters to 2 minutes.
2012-06-29 16:25:40 -07:00
Matei Zaharia
08c50ad1fc
Added script for launching Spark on EC2 from Mesos, to make it easier
...
for new users to get up and running on EC2.
2012-06-10 23:06:15 -07:00