ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1426 commits 6 branches 181 tags 397 MiB
Commit graph

1426 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 8fd374dfb2 Merge pull request #325 from tomdz/master 
Mark hadoop dependencies provided in all library artifacts
 2012-12-13 12:44:20 -08:00
Matei Zaharia 391e5a194a Merge pull request #323 from admobius/master 
SPARK-626: Remove security groups properly
 2012-12-11 12:01:42 -08:00
Peter Sankauskas f97ce3ae14 SPARK-626: Making security group deletion optional, handling retried 
when deleting security groups fails, fixing bug when using all zones but
only 1 slave.
 2012-12-11 10:48:21 -08:00
Thomas Dudziak 02d64f9662 Mark hadoop dependencies provided in all library artifacts 2012-12-10 21:27:54 -08:00
Matei Zaharia 75b8d1f870 Merge pull request #324 from rxin/master 
Suppress shuffle block updates when a slave node comes back.
 2012-12-10 20:39:08 -08:00
Reynold Xin 21b271f5bd Suppress shuffle block updates when a slave node comes back. 2012-12-10 20:36:03 -08:00
Peter Sankauskas 9f964612a1 SPARK-626: Remove rules before removing security groups, with a pause in 
between so wait for AWS eventual consistency to catch up.
 2012-12-10 17:44:09 -08:00
Matei Zaharia c10b229992 Merge pull request #319 from mbautin/cdh4.1.2 
Bump CDH version for the Hadoop 2 profile to 4.1.2
 2012-12-10 15:21:01 -08:00
Matei Zaharia 1ae8978298 Merge pull request #320 from tomdz/master 
Shaded repl jar for hadoop1 profile needs to include hadoop classes
 2012-12-10 15:15:10 -08:00
Matei Zaharia 01c1f97e95 Make "run" script work with Maven builds 2012-12-10 15:13:16 -08:00
Matei Zaharia 597520ae20 Make sure the SSH key we copy to EC2 has permissions 600. 
SPARK-539 #resolve
 2012-12-10 15:12:06 -08:00
Thomas Dudziak c1d15ae3d5 Shaded repl jar for hadoop1 profile needs to include hadoop classes 2012-12-10 15:06:28 -08:00
Mikhail Bautin 450659079a Bump CDH version for the Hadoop 2 profile to 4.1.2 2012-12-10 11:27:20 -08:00
Matei Zaharia a1a2daa7ef Merge pull request #317 from woggling/block-manager-heartbeat 
Implement block manager heartbeat
 2012-12-10 11:03:55 -08:00
Matei Zaharia a9ea14d6e7 Merge pull request #318 from tomdz/master 
Minor tweaks to the debian build
 2012-12-10 10:59:41 -08:00
Matei Zaharia ccff0a089a Use the same output directories that SBT had in subprojects 
This will make it easier to make the "run" script work with a Maven build
 2012-12-10 10:58:56 -08:00
Thomas Dudziak 0e5b1f7981 Minor tweaks to the debian build 2012-12-10 10:30:30 -08:00
Charles Reiss b6b62d774f Decrease BlockManagerMaster logging verbosity 2012-12-10 00:31:55 -08:00
Charles Reiss 5d3e917d09 Use Akka scheduler for BlockManager heart beats. 
Adds required ActorSystem argument to BlockManager constructors.
 2012-12-10 00:31:50 -08:00
Charles Reiss b53dd28c90 Changed default block manager heartbeat interval to 5 s 2012-12-09 23:03:34 -08:00
Matei Zaharia beb440089e Merge pull request #310 from tomdz/master-mavenized 
Maven build setup
 2012-12-09 21:40:05 -08:00
Matei Zaharia e1d7cd2276 Search for a non-loopback address in Utils.getLocalIpAddress 2012-12-08 00:33:11 -08:00
Charles Reiss 714c8d32d5 Don't divide by milliseconds by 1000 more. 2012-12-06 18:38:34 -08:00
Charles Reiss 8f0819520c map -> foreach 2012-12-06 18:29:50 -08:00
Charles Reiss 7a033fd795 Make LocalSparkCluster use distinct IPs 2012-12-06 00:03:08 -08:00
Charles Reiss a2a94fdbc7 Tests for block manager heartbeats. 2012-12-05 23:36:05 -08:00
Charles Reiss d21ca010ac Add block manager heart beats. 
Renames old message called 'HeartBeat' to 'BlockUpdate'.

The BlockManager periodically sends a heart beat message to the master.
If the manager is currently not registered. The master responds to the
heart beat by indicating whether the BlockManager is currently registered
with the master. Additionally, the master now also responds to block
updates by indicating whether the BlockManager in question is registered.
When the BlockManager detects (by heart beat or failed block update)
that it stopped being registered, it reregisters and sends block
updates for all its blocks.
 2012-12-05 23:35:20 -08:00
Charles Reiss c9e54a6755 Track block managers by hostname; handle manager removal. 2012-12-05 23:35:20 -08:00
Charles Reiss 5afa2ee9e9 Actually put millis in _lastSeenMs 2012-12-05 23:35:20 -08:00
Charles Reiss 813ac71459 Don't use bogus port number in notifyADeadHost(). 2012-12-05 23:35:20 -08:00
Matei Zaharia ddf6cd012c Merge pull request #316 from JoshRosen/fix/ec2-web-ui-links 
Use external addresses in standalone web UI when running on EC2
 2012-12-04 15:34:27 -08:00
Josh Rosen cdaa0fad51 Use external addresses in standalone WebUI on EC2. 2012-12-01 18:19:13 -08:00
Matei Zaharia 8d3713c221 Merge pull request #314 from pwendell/quickstart-bugfix 
Adding multi-jar constructor in quickstart
 2012-11-28 22:59:28 -08:00
Thomas Dudziak 84e584fa8c Code review feedback fix 2012-11-28 19:46:06 -08:00
Patrick Wendell 6ceb559994 Adding multi-jar constructor in quickstart 2012-11-27 23:32:24 -08:00
Matei Zaharia f86960cba9 Merge pull request #313 from rxin/pde_size_compress 
Added a partition preserving flag to MapPartitionsWithSplitRDD.
 2012-11-27 22:39:25 -08:00
Matei Zaharia 3ebd8e1885 Added zip to Java API 2012-11-27 22:38:09 -08:00
Matei Zaharia 27e43abd19 Added a zip() operation for RDDs with the same shape (number of 
partitions and number of elements in each partition)
 2012-11-27 22:27:47 -08:00
Matei Zaharia 59c0a9ad16 Use hostname instead of IP in deploy scripts to let Akka connect properly 2012-11-27 21:00:04 -08:00
Matei Zaharia f410a111ad Merge branch 'master' of github.com:mesos/spark 2012-11-27 20:51:58 -08:00
Josh Rosen 7d71b9a56a Fix NullPointerException caused by unregistered map outputs. 2012-11-27 20:51:51 -08:00
Matei Zaharia 935c468b71 Merge pull request #311 from woggling/map-output-npe 
Fix NullPointerException when map output unregistered from MapOutputTracker twice
 2012-11-27 20:50:48 -08:00
Reynold Xin bd6dd1a3a6 Added a partition preserving flag to MapPartitionsWithSplitRDD. 2012-11-27 19:43:30 -08:00
Matei Zaharia 60cb3e9380 Merge pull request #312 from rxin/pde_size_compress 
For size compression, compress non zero values into non zero values.
 2012-11-27 19:25:45 -08:00
Reynold Xin f24bfd2dd1 For size compression, compress non zero values into non zero values. 2012-11-27 19:20:45 -08:00
Thomas Dudziak 3b643e86bc Updated versions in the pom.xml files to match current master 2012-11-27 17:50:42 -08:00
Charles Reiss cf79de425d Fix NullPointerException when unregistering a map output twice. 2012-11-27 16:12:05 -08:00
Charles Reiss 5fa868b98b Tests for MapOutputTracker. 2012-11-27 16:05:36 -08:00
Thomas Dudziak 69297c64be Addressed code review comments 2012-11-27 15:45:16 -08:00
Thomas Dudziak 24e1e425cd Include the configuration templates in the debian package 2012-11-20 16:19:56 -08:00