Matei Zaharia
1ae8978298
Merge pull request #320 from tomdz/master
...
Shaded repl jar for hadoop1 profile needs to include hadoop classes
2012-12-10 15:15:10 -08:00
Matei Zaharia
01c1f97e95
Make "run" script work with Maven builds
2012-12-10 15:13:16 -08:00
Matei Zaharia
597520ae20
Make sure the SSH key we copy to EC2 has permissions 600.
...
SPARK-539 #resolve
2012-12-10 15:12:06 -08:00
Thomas Dudziak
c1d15ae3d5
Shaded repl jar for hadoop1 profile needs to include hadoop classes
2012-12-10 15:06:28 -08:00
Mikhail Bautin
450659079a
Bump CDH version for the Hadoop 2 profile to 4.1.2
2012-12-10 11:27:20 -08:00
Matei Zaharia
a1a2daa7ef
Merge pull request #317 from woggling/block-manager-heartbeat
...
Implement block manager heartbeat
2012-12-10 11:03:55 -08:00
Matei Zaharia
a9ea14d6e7
Merge pull request #318 from tomdz/master
...
Minor tweaks to the debian build
2012-12-10 10:59:41 -08:00
Matei Zaharia
ccff0a089a
Use the same output directories that SBT had in subprojects
...
This will make it easier to make the "run" script work with a Maven build
2012-12-10 10:58:56 -08:00
Thomas Dudziak
0e5b1f7981
Minor tweaks to the debian build
2012-12-10 10:30:30 -08:00
Charles Reiss
b6b62d774f
Decrease BlockManagerMaster logging verbosity
2012-12-10 00:31:55 -08:00
Charles Reiss
5d3e917d09
Use Akka scheduler for BlockManager heart beats.
...
Adds required ActorSystem argument to BlockManager constructors.
2012-12-10 00:31:50 -08:00
Charles Reiss
b53dd28c90
Changed default block manager heartbeat interval to 5 s
2012-12-09 23:03:34 -08:00
Matei Zaharia
beb440089e
Merge pull request #310 from tomdz/master-mavenized
...
Maven build setup
2012-12-09 21:40:05 -08:00
Matei Zaharia
e1d7cd2276
Search for a non-loopback address in Utils.getLocalIpAddress
2012-12-08 00:33:11 -08:00
Charles Reiss
714c8d32d5
Don't divide by milliseconds by 1000 more.
2012-12-06 18:38:34 -08:00
Charles Reiss
8f0819520c
map -> foreach
2012-12-06 18:29:50 -08:00
Charles Reiss
7a033fd795
Make LocalSparkCluster use distinct IPs
2012-12-06 00:03:08 -08:00
Charles Reiss
a2a94fdbc7
Tests for block manager heartbeats.
2012-12-05 23:36:05 -08:00
Charles Reiss
d21ca010ac
Add block manager heart beats.
...
Renames old message called 'HeartBeat' to 'BlockUpdate'.
The BlockManager periodically sends a heart beat message to the master.
If the manager is currently not registered. The master responds to the
heart beat by indicating whether the BlockManager is currently registered
with the master. Additionally, the master now also responds to block
updates by indicating whether the BlockManager in question is registered.
When the BlockManager detects (by heart beat or failed block update)
that it stopped being registered, it reregisters and sends block
updates for all its blocks.
2012-12-05 23:35:20 -08:00
Charles Reiss
c9e54a6755
Track block managers by hostname; handle manager removal.
2012-12-05 23:35:20 -08:00
Charles Reiss
5afa2ee9e9
Actually put millis in _lastSeenMs
2012-12-05 23:35:20 -08:00
Charles Reiss
813ac71459
Don't use bogus port number in notifyADeadHost().
2012-12-05 23:35:20 -08:00
Matei Zaharia
ddf6cd012c
Merge pull request #316 from JoshRosen/fix/ec2-web-ui-links
...
Use external addresses in standalone web UI when running on EC2
2012-12-04 15:34:27 -08:00
Josh Rosen
cdaa0fad51
Use external addresses in standalone WebUI on EC2.
2012-12-01 18:19:13 -08:00
Matei Zaharia
8d3713c221
Merge pull request #314 from pwendell/quickstart-bugfix
...
Adding multi-jar constructor in quickstart
2012-11-28 22:59:28 -08:00
Thomas Dudziak
84e584fa8c
Code review feedback fix
2012-11-28 19:46:06 -08:00
Patrick Wendell
6ceb559994
Adding multi-jar constructor in quickstart
2012-11-27 23:32:24 -08:00
Matei Zaharia
f86960cba9
Merge pull request #313 from rxin/pde_size_compress
...
Added a partition preserving flag to MapPartitionsWithSplitRDD.
2012-11-27 22:39:25 -08:00
Matei Zaharia
3ebd8e1885
Added zip to Java API
2012-11-27 22:38:09 -08:00
Matei Zaharia
27e43abd19
Added a zip() operation for RDDs with the same shape (number of
...
partitions and number of elements in each partition)
2012-11-27 22:27:47 -08:00
Matei Zaharia
59c0a9ad16
Use hostname instead of IP in deploy scripts to let Akka connect properly
2012-11-27 21:00:04 -08:00
Matei Zaharia
f410a111ad
Merge branch 'master' of github.com:mesos/spark
2012-11-27 20:51:58 -08:00
Josh Rosen
7d71b9a56a
Fix NullPointerException caused by unregistered map outputs.
2012-11-27 20:51:51 -08:00
Matei Zaharia
935c468b71
Merge pull request #311 from woggling/map-output-npe
...
Fix NullPointerException when map output unregistered from MapOutputTracker twice
2012-11-27 20:50:48 -08:00
Reynold Xin
bd6dd1a3a6
Added a partition preserving flag to MapPartitionsWithSplitRDD.
2012-11-27 19:43:30 -08:00
Matei Zaharia
60cb3e9380
Merge pull request #312 from rxin/pde_size_compress
...
For size compression, compress non zero values into non zero values.
2012-11-27 19:25:45 -08:00
Reynold Xin
f24bfd2dd1
For size compression, compress non zero values into non zero values.
2012-11-27 19:20:45 -08:00
Thomas Dudziak
3b643e86bc
Updated versions in the pom.xml files to match current master
2012-11-27 17:50:42 -08:00
Charles Reiss
cf79de425d
Fix NullPointerException when unregistering a map output twice.
2012-11-27 16:12:05 -08:00
Charles Reiss
5fa868b98b
Tests for MapOutputTracker.
2012-11-27 16:05:36 -08:00
Thomas Dudziak
69297c64be
Addressed code review comments
2012-11-27 15:45:16 -08:00
Thomas Dudziak
24e1e425cd
Include the configuration templates in the debian package
2012-11-20 16:19:56 -08:00
Thomas Dudziak
811a32257b
Added maven and debian build files
2012-11-20 16:19:51 -08:00
Matei Zaharia
cd16eab0db
Merge pull request #309 from admobius/use-boto-config-file
...
Improved use of Boto
2012-11-19 15:51:41 -08:00
Peter Sankauskas
dc2fb3c4b6
Allow Boto to use the other config options it supports, and gracefully
...
handling Boto connection exceptions (like AuthFailure)
2012-11-19 14:21:16 -08:00
Matei Zaharia
85ce5f27c1
Merge pull request #308 from admobius/multi-zone
...
Let EC2 script launch slaves in multiple availability zones
2012-11-19 13:24:09 -08:00
Matei Zaharia
3ff6f4bdee
Merge pull request #304 from mbautin/configurable_local_ip
...
SPARK-624: make the default local IP customizable
2012-11-19 13:23:39 -08:00
mbautin
00f4e3ff9c
Addressing Matei's comment: SPARK_LOCAL_IP environment variable
2012-11-19 11:52:10 -08:00
Peter Sankauskas
606d252d26
Adding comment about additional bandwidth charges
2012-11-17 23:09:11 -08:00
Matei Zaharia
20a1058dd5
Merge pull request #305 from woggling/exit-on-uncaught
...
Set default uncaught exception handler to exit
2012-11-16 20:56:09 -08:00