Matei Zaharia
af74217c96
Undo JLine fix that turns out to only be needed when buildr is running
...
on JRuby. This is quite ugly: JRuby has its own version of JLine which
is older than Scala's, and JLine changed API in such a way that code
written for the new version won't compile with the old one and vice
versa. Sadly, this might be a reason to drop buildr, unless we can
package a JRuby with it that uses the right version, or we ask people to
only use the C Ruby version of buildr (which doesn't work on OS X right
now!)
2010-11-13 18:38:50 -08:00
Matei Zaharia
b84769a107
Modified project structure to work with buildr
2010-11-13 17:18:05 -08:00
Matei Zaharia
f8966ffc11
Added a shuffle test with negative hash codes for some keys (this was a bug earlier)
2010-11-12 16:18:45 -08:00
Matei Zaharia
d0a9966555
Unit tests for shuffle operations. Fixes #33 .
2010-11-12 16:12:14 -08:00
Mosharaf Chowdhury
9915fd22c8
Added BroadcastBT.EndGameFraction config option to control when to start end game.
2010-11-12 15:27:47 -08:00
Mosharaf Chowdhury
c6874d4370
Talk to peers even if current knowledge says they dont have anything to send.
2010-11-12 14:36:34 -08:00
Mosharaf Chowdhury
0adb12e5aa
Merge branch 'mos-bt' into mos-bt-ask-for-block
2010-11-11 14:07:02 -08:00
Mosharaf Chowdhury
23ebac8c30
This reverts the addToListOfSources methods of commit 642098c211.
2010-11-11 14:05:38 -08:00
Mosharaf Chowdhury
88d08b7a89
Bug fix.
2010-11-10 17:04:14 -08:00
Mosharaf Chowdhury
eb70f6a36e
Receivers explicitly ask for blocks to send.
...
The Random generator must be updated so that everyone does not ask for the same block during the initial phase.
2010-11-10 16:47:45 -08:00
Mosharaf Chowdhury
d7579e743a
Style update: all functions that return void should have a Unit return type.
2010-11-10 10:32:29 -08:00
Mosharaf Chowdhury
642098c211
Bug fixes and/or minor optimization.
2010-11-09 23:59:06 -08:00
Mosharaf Chowdhury
b58bbadbfb
All daemon => Fast shutdown :)
2010-11-09 17:05:19 -08:00
Mosharaf Chowdhury
76610e842b
Removed some unnecessary stuff.
2010-11-09 16:59:32 -08:00
Mosharaf Chowdhury
50612ab3a9
Every thread is a daemon thread => Program shuts down ASAP once its done.
...
Some log messages added to blank exception handlers.
2010-11-09 16:44:42 -08:00
Matei Zaharia
7b25ab87af
Added options for using an external HTTP server with LocalFileShuffle
2010-11-09 13:46:30 -08:00
Mosharaf Chowdhury
1820634dbf
Bug fix: java-opts had wrong (non-capitalized) variable names.
2010-11-08 18:32:17 -08:00
Matei Zaharia
504f839c65
Removed unnecessary collectAsMap
2010-11-08 08:49:42 -08:00
Matei Zaharia
9d3f05a990
Made shuffle algorithm pluggable and added LocalFileShuffle.
2010-11-08 00:46:12 -08:00
Mosharaf Chowdhury
6cb76d9708
Commented out some cluttering log messages.
2010-11-07 19:16:03 -08:00
Mosharaf Chowdhury
cbce44304c
Added exception handling to 'sendStopBroadcastNotifications'
2010-11-06 19:37:27 -07:00
Mosharaf Chowdhury
49a2db09fb
Graceful shutdown is working with dualMode=false. Probably will have to remove dualMode completely.
...
Made BroadcastCS code more consistent with BT branches.
2010-11-06 19:27:46 -07:00
Mosharaf Chowdhury
c37c74919f
Bug fix + minor changes
2010-11-06 16:54:55 -07:00
Mosharaf Chowdhury
cbb29fae1a
Updated log outputs for consistency with BT branches.
...
Code formatting.
2010-11-06 16:52:50 -07:00
Matei Zaharia
d9ea6d69a5
Create output files one by one instead of at the same time in the map
...
phase of DfsShuffle.
2010-11-06 10:53:57 -07:00
Mosharaf Chowdhury
878d157ce3
Graceful shutdown after a single transmission in the swarm is over.
...
There might still be a problem with the Tracker shutdown. It must be done explicitly by SparkContext.
2010-11-04 22:09:14 -07:00
Matei Zaharia
16ff4dc0be
Merge branch 'matei-shuffle' of github.com:mesos/spark into matei-shuffle
2010-11-04 14:40:36 -07:00
Matei Zaharia
d984b8ab23
Properly set the number of output splits in DFS shuffle
2010-11-04 14:39:55 -07:00
root
4cc0984b43
Fixed a small bug in DFS shuffle -- the number of reduce tasks was not being set based on numOutputSplits
2010-11-04 21:34:55 +00:00
Matei Zaharia
96f0be935a
Added groupBy function in RDD
2010-11-03 23:58:53 -07:00
Matei Zaharia
72ec298cd4
Added reduceByKey, groupByKey and join operations based on combine, as
...
well as versions of the shuffle operations that set the number of splits
automatically.
2010-11-03 23:51:11 -07:00
Matei Zaharia
d947cb9778
Fixed a bug with negative hashcodes
2010-11-03 22:52:41 -07:00
Matei Zaharia
44530c310b
Made DFS shuffle's "reduce tasks" fetch inputs in a random order so they
...
don't all hit the same nodes at the same time.
2010-11-03 22:45:44 -07:00
Matei Zaharia
820dac5afe
Initial work towards a simple HDFS-based shuffle.
2010-11-03 21:27:24 -07:00
Matei Zaharia
648f42933a
Made alltests write test output as XML in build/test_results
2010-11-02 12:53:38 -07:00
Matei Zaharia
6f93baa463
'Running on Mesos' test is now only run when MESOS_HOME is set
2010-11-02 12:51:22 -07:00
Mosharaf Chowdhury
10fc66b1c4
Print reception messages for duplicate blocks as well...
2010-11-01 14:00:40 -07:00
Mosharaf Chowdhury
03d73dea9f
Refactoring...
2010-10-28 15:09:58 -07:00
Mosharaf Chowdhury
681dba1373
Removed some deprecated speed related code.
2010-10-28 14:40:49 -07:00
Mosharaf Chowdhury
bcc7d38ace
Added SpeedTracker to keep track of reception time per block from each source a peer has ever talked to.
2010-10-28 14:37:59 -07:00
Mosharaf Chowdhury
44eb79f64a
- Measuring approximate time to receive a block from the sending peer.
...
- Added some Exception handling/suppression code.
2010-10-27 17:41:37 -07:00
Mosharaf Chowdhury
ae2c93a54b
Fixed some configuration bugs
2010-10-27 16:18:26 -07:00
Mosharaf Chowdhury
e4dc7847ba
- Receivers now send back the latest hasBlocksBitVector after every reception
...
- Will probably settle for the push model. Still not removing the pull related codes
- Refactoring...
2010-10-27 15:36:58 -07:00
Mosharaf Chowdhury
059b6ae475
Added data structures to turn transmission to a pull model from a push one.
2010-10-27 15:05:29 -07:00
Mosharaf Chowdhury
f7364111bb
TalkToGuide breaks once the peer has all the blocks.
2010-10-26 21:26:45 -07:00
Mosharaf Chowdhury
8587353b3e
Added some tuning parameters.
2010-10-26 21:16:37 -07:00
Mosharaf Chowdhury
d70f3549ff
MaxTxPeers and MaxRxPeers set upper limit on concurrent sending and receiving threads for every peer.
...
Relevant timeout code is also in. There is still an exception to take care of.
2010-10-26 17:53:03 -07:00
Mosharaf Chowdhury
1877f1c38d
No one is waiting on hasBlocksLock. Removed...
2010-10-26 16:18:13 -07:00
Mosharaf Chowdhury
bc1c51817c
Beautified code...
2010-10-26 16:08:22 -07:00
Mosharaf Chowdhury
d673be0e06
Minor changes. Still need to run on Mesos+EC2
2010-10-26 15:07:03 -07:00
