ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
11343 commits 6 branches 181 tags 397 MiB
Commit graph

177 commits

Author SHA1 Message Date
Patrick Wendell c02585ea13 Make initial connection failure message less daunting. 
Right now it seems like something has gone wrong when this message is printed out.
Instead, this is a normal condition. So I changed the message a bit.
 2013-08-21 15:45:45 -07:00
Patrick Wendell 6be6b71c8c Merge branch 'master' into ec2-updates 
Conflicts:
	ec2/spark_ec2.py
 2013-08-21 15:34:31 -07:00
Patrick Wendell 51a1a0c602 Bump spark version 2013-08-20 22:14:52 -07:00
Patrick Wendell 07e5c8b695 Set default Hadoop version to 1 2013-08-20 15:49:52 -07:00
Matei Zaharia 793a722f8e Allow some wiggle room in UISuite port test and in EC2 ports 2013-08-19 18:51:00 -07:00
Matei Zaharia 498a26189b Small fixes to web UI: 
- Use SPARK_PUBLIC_DNS environment variable if set (for EC2)
- Use a non-ephemeral port (3030 instead of 33000) by default
- Updated test to use non-ephemeral port too
 2013-08-19 18:17:49 -07:00
Patrick Wendell 7e1b7d8a96 Adding web UI port 2013-08-08 14:17:11 -07:00
Patrick Wendell 1105766295 Point to new repository/branch 2013-08-01 15:43:41 -07:00
Patrick Wendell 56715587e4 Pointing to new EC2 repository 2013-08-01 12:27:43 -07:00
Patrick Wendell 5cc725a0e3 Merge branch 'master' into ec2-updates 
Conflicts:
	ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh
 2013-07-31 21:35:12 -07:00
Patrick Wendell 893aaff738 Don't install MapReduce for Hadoop1 2013-07-30 14:55:04 -07:00
Matei Zaharia 207548b67b Open up Job UI ports (33000-33010) on EC2 clusters 2013-07-29 17:19:33 -07:00
Ubuntu 88a0823c58 Consistently invoke bash with /usr/bin/env bash in scripts to make code more portable (JIRA Ticket SPARK-817) 2013-07-18 00:51:18 +00:00
Patrick Wendell 31b3c1df54 Small improvement in python script 2013-07-17 17:43:15 -07:00
Matei Zaharia af3c9d5042 Add Apache license headers and LICENSE and NOTICE files 2013-07-16 17:21:33 -07:00
Patrick Wendell 8dccee16af Bug fix 2013-07-11 16:50:27 -07:00
Patrick Wendell 31c18a2528 Removing support for custom Shark version 2013-07-11 16:18:16 -07:00
Patrick Wendell 13809d363c Small fix 2013-07-11 15:45:35 -07:00
Patrick Wendell f1689185d6 Adding ability to custom deploy only Spark 2013-07-11 15:44:31 -07:00
Patrick Wendell 311e63f2c7 Removing mesos terminology 2013-07-11 14:13:10 -07:00
Patrick Wendell 13e3dd98d8 Removing mesos support 2013-06-26 11:36:39 -07:00
Patrick Wendell d5f74aa9b7 Cleanup of AMI tag 2013-06-13 10:30:59 -07:00
Patrick Wendell eae7590d0d Support for multiple hadoop versions 2013-06-12 18:39:01 -07:00
Patrick Wendell bb328c8ff8 Re-working spark versions to use templates 2013-06-11 20:14:33 -07:00
Patrick Wendell 2124563a00 Remving support for old scripts 2013-06-11 17:52:10 -07:00
Patrick Wendell 8637c02191 Minor doc cleanup 2013-06-07 14:54:26 -07:00
Patrick Wendell 0f4c0e461f Removing now defunct ami documentation 2013-06-07 14:39:58 -07:00
Patrick Wendell d65d3dd8f0 Fixing syntax error 2013-06-07 14:16:01 -07:00
Patrick Wendell 2477736237 Adding spark shark modules 2013-06-07 11:55:46 -07:00
Patrick Wendell db50379737 Decoupling spark version from ami 2013-06-07 11:54:16 -07:00
Patrick Wendell a07bd286bb Some changes in response to Josh 2013-05-18 16:00:32 -07:00
Patrick Wendell e8a1d02956 Adding support for tag 2013-05-10 15:13:22 -07:00
Patrick Wendell 2197b2159a Removing unnecessary parsing 2013-05-09 11:27:13 -07:00
Patrick Wendell 36c117a1f1 Slight change to AMI versioning 2013-05-08 22:50:53 -07:00
Patrick Wendell f6c965a98b Changing spark version and availability zone fix 2013-05-08 21:46:01 -07:00
Patrick Wendell 976e02491e Resolve AMI region automatically 2013-05-08 21:46:01 -07:00
Patrick Wendell f8213aab4b Adding automatic resolve of AMI 2013-05-08 21:46:01 -07:00
Patrick Wendell 03b58378d0 Adding mapreduce module 2013-05-08 21:46:01 -07:00
Patrick Wendell 73f8cd237e Adding extra instance types 2013-05-08 21:46:01 -07:00
Patrick Wendell 9ff8ad4bec REVERT ME: Linking to Patrick's ec2 repo for now 2013-05-08 21:46:00 -07:00
Josh Rosen e014c1d1cb Fix SPARK-670: EC2 start command should require -i option. 2013-05-05 11:30:36 -07:00
Holden Karau 9bf24e1d61 Just use a loop for retries 2013-04-09 21:37:02 -07:00
Holden Karau ff2130a0ad Retry failed ssh commands. This is especially useful during system startup when the hosts may not have yet come on-line but can be useful at other points for people with flakey connections 2013-04-06 00:35:50 -07:00
Matei Zaharia baa30fcd99 Use new Spark EC2 scripts by default 2013-02-26 23:38:50 -08:00
Shivaram Venkataraman 6cba5a48b0 Print cluster url after setup completes 2013-02-18 18:30:36 -08:00
Shivaram Venkataraman e7cdf7a6a4 Print ganglia url after setup 2013-02-18 17:15:22 -08:00
Shivaram Venkataraman 03f45a18d5 Use port 5080 for httpd/ganglia 2013-02-18 16:56:01 -08:00
Matei Zaharia 455d015076 Clean up EC2 script options a bit 2013-02-17 16:53:12 -08:00
Matei Zaharia 08e444df0e Change EC2 script to use 0.6 AMIs by default, for now 2013-02-17 14:01:48 -08:00
Shivaram Venkataraman bf675ab4f9 Turn on ganglia by default 2013-01-31 21:43:45 -08:00
Shivaram Venkataraman da44a391fb Add an option to use the old scripts 2013-01-28 11:16:14 -08:00
Shivaram Venkataraman dc9d3ab6ed Add option to start ganglia. Also enable Hadoop ports even if cluster type is 
not mesos
 2013-01-27 00:26:00 -08:00
Shivaram Venkataraman 0243b081ce Fix swap variable name 2013-01-27 00:25:29 -08:00
Shivaram Venkataraman d012cfa0e8 Update spark_ec2.py to use new spark-ec2 scripts 2013-01-26 22:48:39 -08:00
Matei Zaharia 391e5a194a Merge pull request #323 from admobius/master 
SPARK-626: Remove security groups properly
 2012-12-11 12:01:42 -08:00
Peter Sankauskas f97ce3ae14 SPARK-626: Making security group deletion optional, handling retried 
when deleting security groups fails, fixing bug when using all zones but
only 1 slave.
 2012-12-11 10:48:21 -08:00
Peter Sankauskas 9f964612a1 SPARK-626: Remove rules before removing security groups, with a pause in 
between so wait for AWS eventual consistency to catch up.
 2012-12-10 17:44:09 -08:00
Matei Zaharia 597520ae20 Make sure the SSH key we copy to EC2 has permissions 600. 
SPARK-539 #resolve
 2012-12-10 15:12:06 -08:00
Matei Zaharia cd16eab0db Merge pull request #309 from admobius/use-boto-config-file 
Improved use of Boto
 2012-11-19 15:51:41 -08:00
Peter Sankauskas dc2fb3c4b6 Allow Boto to use the other config options it supports, and gracefully 
handling Boto connection exceptions (like AuthFailure)
 2012-11-19 14:21:16 -08:00
Matei Zaharia 85ce5f27c1 Merge pull request #308 from admobius/multi-zone 
Let EC2 script launch slaves in multiple availability zones
 2012-11-19 13:24:09 -08:00
Peter Sankauskas 606d252d26 Adding comment about additional bandwidth charges 2012-11-17 23:09:11 -08:00
Peter Sankauskas 32442ee1e1 Giving the Spark EC2 script the ability to launch instances spread 
across multiple availability zones in order to make the cluster more
resilient to failure
 2012-11-16 17:25:28 -08:00
Peter Sankauskas 6d22f7ccb8 Delete security groups when deleting the cluster. As many operations 
are done on instances in specific security groups, this seems like a
reasonable thing to clean up.
 2012-11-16 14:02:43 -08:00
Josh Rosen 594eed31c4 Fix check for existing instances during EC2 launch. 2012-11-03 17:02:47 -07:00
Josh Rosen 96c9bcfd8d Cancel spot instance requests when exiting spark-ec2. 2012-10-30 23:32:38 -07:00
Josh Rosen 365a4c1e68 Allow EC2 script to stop/destroy cluster after master/slave failures. 2012-10-18 10:36:50 -07:00
Matei Zaharia 5b7ee173e1 Update EC2 scripts for Spark 0.6 2012-10-12 19:53:03 -07:00
Ray Racine c52bd1b6b1 Add m1.medium node option to cluster management script. 2012-10-09 11:05:13 -04:00
Matei Zaharia 47507d69d9 Made region used by spark-ec2 configurable. 2012-08-28 22:40:00 -07:00
Denny 8dc7242544 Use root login in standalone AMI 2012-08-14 10:18:24 -07:00
Denny 7152c7c12d rsync root directory in EC2 script 2012-08-14 09:26:47 -07:00
Matei Zaharia 8069bd5b41 Removed separate launcher for EC2 standalone cluster 2012-08-09 22:45:24 +02:00
Denny aaed039e36 Merged standalone and mesos EC2 scripts 2012-08-02 15:23:52 -07:00
Denny 0ee44c225e Spark standalone mode cluster scripts. 
Heavily inspired by Hadoop cluster scripts ;-)
 2012-08-01 20:38:52 -07:00
Matei Zaharia 8a5bd4e765 Increase the default wait time for EC2 clusters to 2 minutes. 2012-06-29 16:25:40 -07:00
Matei Zaharia 08c50ad1fc Added script for launching Spark on EC2 from Mesos, to make it easier 
for new users to get up and running on EC2.
 2012-06-10 23:06:15 -07:00