ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2656 commits 6 branches 181 tags 397 MiB
Commit graph

47 commits

Author SHA1 Message Date
Patrick Wendell db50379737 Decoupling spark version from ami 2013-06-07 11:54:16 -07:00
Patrick Wendell a07bd286bb Some changes in response to Josh 2013-05-18 16:00:32 -07:00
Patrick Wendell e8a1d02956 Adding support for tag 2013-05-10 15:13:22 -07:00
Patrick Wendell 2197b2159a Removing unnecessary parsing 2013-05-09 11:27:13 -07:00
Patrick Wendell 36c117a1f1 Slight change to AMI versioning 2013-05-08 22:50:53 -07:00
Patrick Wendell f6c965a98b Changing spark version and availability zone fix 2013-05-08 21:46:01 -07:00
Patrick Wendell 976e02491e Resolve AMI region automatically 2013-05-08 21:46:01 -07:00
Patrick Wendell f8213aab4b Adding automatic resolve of AMI 2013-05-08 21:46:01 -07:00
Patrick Wendell 03b58378d0 Adding mapreduce module 2013-05-08 21:46:01 -07:00
Patrick Wendell 73f8cd237e Adding extra instance types 2013-05-08 21:46:01 -07:00
Patrick Wendell 9ff8ad4bec REVERT ME: Linking to Patrick's ec2 repo for now 2013-05-08 21:46:00 -07:00
Holden Karau 9bf24e1d61 Just use a loop for retries 2013-04-09 21:37:02 -07:00
Holden Karau ff2130a0ad Retry failed ssh commands. This is especially useful during system startup when the hosts may not have yet come on-line but can be useful at other points for people with flakey connections 2013-04-06 00:35:50 -07:00
Matei Zaharia baa30fcd99 Use new Spark EC2 scripts by default 2013-02-26 23:38:50 -08:00
Shivaram Venkataraman 6cba5a48b0 Print cluster url after setup completes 2013-02-18 18:30:36 -08:00
Shivaram Venkataraman e7cdf7a6a4 Print ganglia url after setup 2013-02-18 17:15:22 -08:00
Shivaram Venkataraman 03f45a18d5 Use port 5080 for httpd/ganglia 2013-02-18 16:56:01 -08:00
Matei Zaharia 455d015076 Clean up EC2 script options a bit 2013-02-17 16:53:12 -08:00
Matei Zaharia 08e444df0e Change EC2 script to use 0.6 AMIs by default, for now 2013-02-17 14:01:48 -08:00
Shivaram Venkataraman bf675ab4f9 Turn on ganglia by default 2013-01-31 21:43:45 -08:00
Shivaram Venkataraman da44a391fb Add an option to use the old scripts 2013-01-28 11:16:14 -08:00
Shivaram Venkataraman dc9d3ab6ed Add option to start ganglia. Also enable Hadoop ports even if cluster type is 
not mesos
 2013-01-27 00:26:00 -08:00
Shivaram Venkataraman 0243b081ce Fix swap variable name 2013-01-27 00:25:29 -08:00
Shivaram Venkataraman d012cfa0e8 Update spark_ec2.py to use new spark-ec2 scripts 2013-01-26 22:48:39 -08:00
Matei Zaharia 391e5a194a Merge pull request #323 from admobius/master 
SPARK-626: Remove security groups properly
 2012-12-11 12:01:42 -08:00
Peter Sankauskas f97ce3ae14 SPARK-626: Making security group deletion optional, handling retried 
when deleting security groups fails, fixing bug when using all zones but
only 1 slave.
 2012-12-11 10:48:21 -08:00
Peter Sankauskas 9f964612a1 SPARK-626: Remove rules before removing security groups, with a pause in 
between so wait for AWS eventual consistency to catch up.
 2012-12-10 17:44:09 -08:00
Matei Zaharia 597520ae20 Make sure the SSH key we copy to EC2 has permissions 600. 
SPARK-539 #resolve
 2012-12-10 15:12:06 -08:00
Matei Zaharia cd16eab0db Merge pull request #309 from admobius/use-boto-config-file 
Improved use of Boto
 2012-11-19 15:51:41 -08:00
Peter Sankauskas dc2fb3c4b6 Allow Boto to use the other config options it supports, and gracefully 
handling Boto connection exceptions (like AuthFailure)
 2012-11-19 14:21:16 -08:00
Matei Zaharia 85ce5f27c1 Merge pull request #308 from admobius/multi-zone 
Let EC2 script launch slaves in multiple availability zones
 2012-11-19 13:24:09 -08:00
Peter Sankauskas 606d252d26 Adding comment about additional bandwidth charges 2012-11-17 23:09:11 -08:00
Peter Sankauskas 32442ee1e1 Giving the Spark EC2 script the ability to launch instances spread 
across multiple availability zones in order to make the cluster more
resilient to failure
 2012-11-16 17:25:28 -08:00
Peter Sankauskas 6d22f7ccb8 Delete security groups when deleting the cluster. As many operations 
are done on instances in specific security groups, this seems like a
reasonable thing to clean up.
 2012-11-16 14:02:43 -08:00
Josh Rosen 594eed31c4 Fix check for existing instances during EC2 launch. 2012-11-03 17:02:47 -07:00
Josh Rosen 96c9bcfd8d Cancel spot instance requests when exiting spark-ec2. 2012-10-30 23:32:38 -07:00
Josh Rosen 365a4c1e68 Allow EC2 script to stop/destroy cluster after master/slave failures. 2012-10-18 10:36:50 -07:00
Matei Zaharia 5b7ee173e1 Update EC2 scripts for Spark 0.6 2012-10-12 19:53:03 -07:00
Ray Racine c52bd1b6b1 Add m1.medium node option to cluster management script. 2012-10-09 11:05:13 -04:00
Matei Zaharia 47507d69d9 Made region used by spark-ec2 configurable. 2012-08-28 22:40:00 -07:00
Denny 8dc7242544 Use root login in standalone AMI 2012-08-14 10:18:24 -07:00
Denny 7152c7c12d rsync root directory in EC2 script 2012-08-14 09:26:47 -07:00
Matei Zaharia 8069bd5b41 Removed separate launcher for EC2 standalone cluster 2012-08-09 22:45:24 +02:00
Denny aaed039e36 Merged standalone and mesos EC2 scripts 2012-08-02 15:23:52 -07:00
Denny 0ee44c225e Spark standalone mode cluster scripts. 
Heavily inspired by Hadoop cluster scripts ;-)
 2012-08-01 20:38:52 -07:00
Matei Zaharia 8a5bd4e765 Increase the default wait time for EC2 clusters to 2 minutes. 2012-06-29 16:25:40 -07:00
Matei Zaharia 08c50ad1fc Added script for launching Spark on EC2 from Mesos, to make it easier 
for new users to get up and running on EC2.
 2012-06-10 23:06:15 -07:00