Mosharaf Chowdhury
2b946fb2d1
pickBlockRarestFirst and gossips commented OUT for now.
...
Problem with the rarestFirst implemention is that we are picking peers randomly first and then picking blocks from the random peer using rarestFirst. NOT the right away to do it. It should be the other way around.
Problem with gossip is that peers might end up overwriting newer information by older ones. To fix that we either have to have timestamps or must match the bitVectors before overwriting.
2011-02-13 13:53:15 -08:00
Mosharaf Chowdhury
ca2895ebb0
Fix in rarestFirst implemenation.
...
If there are more than one rarest blocks, pick randomly between them (was deterministic before)
2011-02-10 20:37:44 -08:00
Mosharaf Chowdhury
520bbdc7e3
Peers now gossip about their neighbors when they talk.
2011-02-10 20:15:30 -08:00
Matei Zaharia
dc24aecd8f
Close record readers in HadoopFile after finishing a split
2011-02-10 12:07:48 -08:00
Mosharaf Chowdhury
441462bc7f
Fixed some warnings during compilation.
2011-02-09 12:11:43 -08:00
Mosharaf Chowdhury
1a73c0d265
Merged with master. Using sbt.
2011-02-09 10:48:48 -08:00
Mosharaf Chowdhury
495b38658e
Merge branch 'master' into mos-bt
2011-02-09 10:40:23 -08:00
Matei Zaharia
99f3f23efa
Changed default shuffle to LocalFileShuffle because it's way faster for small files
2011-02-08 17:03:03 -08:00
Matei Zaharia
ec28b607fd
Merge branch 'master' into sbt
...
Conflicts:
Makefile
core/src/main/java/spark/compress/lzf/LZF.java
core/src/main/java/spark/compress/lzf/LZFInputStream.java
core/src/main/java/spark/compress/lzf/LZFOutputStream.java
core/src/main/native/spark_compress_lzf_LZF.c
run
2011-02-02 00:25:54 -08:00
Matei Zaharia
e5c4cd8a5e
Made examples and core subprojects
2011-02-01 15:11:08 -08:00