Mosharaf Chowdhury
c6df327dd7
Updated logging format.
2010-12-04 16:41:13 -08:00
Mosharaf Chowdhury
7df20d681a
Combined MaxRxPeers and MaxTxPeers to a single config parameter MaxConnections
2010-12-04 14:37:16 -08:00
Mosharaf Chowdhury
b1745b3103
Removed an unnecessary byte array in the middle. Probabaly will have to bring it back if we do block level data movement.
2010-12-04 13:55:25 -08:00
Mosharaf Chowdhury
3a671ce989
Config parameters are in place. Good to go (I think)
2010-12-04 10:59:06 -08:00
Mosharaf Chowdhury
476a216d9d
Parallel is working. Need to fix/finalize some config parameters.
2010-12-04 02:05:41 -08:00
Mosharaf Chowdhury
c546c299bc
combining is happening inside the thread. Its still synchronized though.
2010-12-04 00:59:25 -08:00
Mosharaf Chowdhury
0d7ca7751e
Bug fixes. Not yet parallel.
2010-12-04 00:06:47 -08:00
Mosharaf Chowdhury
52086cef32
Building blocks are in place. Still not pulling parallely though.
2010-12-03 20:29:39 -08:00
Mosharaf Chowdhury
540a41163f
UseHttpPipelining is 'true' by default.
2010-12-02 19:56:17 -08:00
Mosharaf Chowdhury
1013a01469
Made the root Broadcast object initilization method 'synchronized'.
2010-12-02 02:36:21 -08:00
Mosharaf Chowdhury
0de859fbe2
Enabling/disabling HTTP pipelining is a config option now. Performance tradeoffs are not obvious yet.
2010-12-02 02:32:44 -08:00
Mosharaf Chowdhury
a69638d98d
Removed deprecated repl file ClassServer.scala
2010-11-30 18:45:04 -08:00
Mosharaf Chowdhury
b7dda4c5bc
Merge branch 'multi-tracker' into mos-bt
...
Conflicts:
conf/java-opts
src/scala/spark/Broadcast.scala
src/scala/spark/DfsBroadcast.scala
src/scala/spark/SparkContext.scala
2010-11-30 18:29:38 -08:00
Mosharaf Chowdhury
815ecd349a
Made Broadcast Pluggable. Finally!
2010-11-30 18:08:49 -08:00
Mosharaf Chowdhury
ea28a30e36
Makes transfer faster for small swarms (with size < MaxRxPeers)
2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury
191af65fb7
- Using the new Cache implementation.
2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury
c9cad03c31
- Using the new Cache implementation.
...
- Removed unused code related to dualMode (deprecated).
2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury
73714da568
Made LocalFileShuffle to be the default.
2010-11-29 19:19:21 -08:00
Mosharaf Chowdhury
c62f2fefa5
Made LocalFileShuffle to be the default.
2010-11-29 19:17:01 -08:00
Mosharaf Chowdhury
15f1f6865d
Variable renaming.
2010-11-29 14:15:36 -08:00
Mosharaf Chowdhury
bb3e7fbf61
Variable renaming.
2010-11-29 14:13:35 -08:00
Mosharaf Chowdhury
1157380f21
Removed unused third party libraries.
2010-11-29 12:21:46 -08:00
Mosharaf Chowdhury
1f7fcb48be
Merge branch 'master' into mos-bt
...
Conflicts:
Makefile
run
src/scala/spark/Broadcast.scala
src/scala/spark/Executor.scala
src/scala/spark/HdfsFile.scala
src/scala/spark/MesosScheduler.scala
src/scala/spark/RDD.scala
src/scala/spark/SparkContext.scala
src/scala/spark/Split.scala
src/scala/spark/Utils.scala
src/scala/spark/repl/SparkInterpreter.scala
third_party/mesos.jar
2010-11-29 12:10:06 -08:00
Mosharaf Chowdhury
9e93f8df12
Moved broadcast implementations to separate files. Ready to merge with 'master'
2010-11-29 11:43:20 -08:00
Mosharaf Chowdhury
8d775448c2
- Resolved some of the simpler TODOs related to different timeout and wait periods.
...
- Removed unused code.
2010-11-29 11:20:04 -08:00
Mosharaf Chowdhury
8494b3a4f9
- Added log messages for benchmarking.
...
- Added GroupByTest.scala for benchmarking.
2010-11-27 23:51:43 -08:00
Mosharaf Chowdhury
19a6b194a9
Merge branch 'master' into multi-tracker
...
Conflicts:
Makefile
run
src/scala/spark/Broadcast.scala
src/scala/spark/Executor.scala
src/scala/spark/HdfsFile.scala
src/scala/spark/MesosScheduler.scala
src/scala/spark/RDD.scala
src/scala/spark/SparkContext.scala
src/scala/spark/Split.scala
src/scala/spark/Utils.scala
src/scala/spark/repl/SparkInterpreter.scala
third_party/mesos.jar
2010-11-27 01:27:20 -08:00
Mosharaf Chowdhury
e4b8db45ae
- Moved DaemonThreadPool factory methods to the Broadcast Object.
...
- Reflection-based broadcast class loading is still not working.
2010-11-27 00:53:57 -08:00
Mosharaf Chowdhury
c25914228c
Moved ChaninedStreaming to a separate file and renamed to ChainedBroadcast.
2010-11-26 23:15:33 -08:00
Mosharaf Chowdhury
98542f81bb
- Removed dual Mode
...
- Removed ununsed speed-related stuff
- Moved (H)DfsBroadcast to its own file
2010-11-26 22:58:35 -08:00
Mosharaf Chowdhury
9d0111659b
Solved compilation issue. Should've been more careful to do a clean compilation before committing.
2010-11-26 22:10:38 -08:00
Mosharaf Chowdhury
9c8a65b920
Removed everything related to SplitStreaming...
2010-11-26 22:06:43 -08:00
Mosharaf Chowdhury
830496b901
Updated some comments.
2010-11-25 23:49:57 -08:00
Mosharaf Chowdhury
690b55917d
Slaves weren't printing hostAddress. Fixed.
2010-11-18 10:53:34 -08:00
Mosharaf Chowdhury
2de1664214
Slaves weren't logging hostAddress. Fixed.
2010-11-18 10:51:57 -08:00
Mosharaf Chowdhury
dfa6540fdd
Everyone logs local hostAddress.
2010-11-18 09:58:46 -08:00
Mosharaf Chowdhury
1f2662d45a
Everyone logs their local hostAddress.
2010-11-18 09:55:19 -08:00
Matei Zaharia
dcfa2ce83b
Further improvements -- build native stuff in target directory and add a
...
test-report target for XML test reports
2010-11-14 00:46:19 -08:00
Matei Zaharia
e86b620f9e
Fixed some more stuff (Eclispe target and native build)
2010-11-13 22:46:00 -08:00
Matei Zaharia
89fcd96702
Initial work to get Spark compiling with SBT 0.7.5 RC0
2010-11-13 22:07:08 -08:00
Mosharaf Chowdhury
c3f354c6fc
Guide now select random peers to select sources instead of rolling statically.
...
Random is good. Deterministic is bad. Well, mostly.
2010-11-13 20:18:48 -08:00
Matei Zaharia
b7574201d5
Added some stuff to .gitignore
2010-11-13 18:41:35 -08:00
Matei Zaharia
f8ea98d989
Remove -unchecked compiler parameter
2010-11-13 18:39:07 -08:00
Matei Zaharia
af74217c96
Undo JLine fix that turns out to only be needed when buildr is running
...
on JRuby. This is quite ugly: JRuby has its own version of JLine which
is older than Scala's, and JLine changed API in such a way that code
written for the new version won't compile with the old one and vice
versa. Sadly, this might be a reason to drop buildr, unless we can
package a JRuby with it that uses the right version, or we ask people to
only use the C Ruby version of buildr (which doesn't work on OS X right
now!)
2010-11-13 18:38:50 -08:00
Matei Zaharia
b84769a107
Modified project structure to work with buildr
2010-11-13 17:18:05 -08:00
Matei Zaharia
f8966ffc11
Added a shuffle test with negative hash codes for some keys (this was a bug earlier)
2010-11-12 16:18:45 -08:00
Matei Zaharia
d0a9966555
Unit tests for shuffle operations. Fixes #33 .
2010-11-12 16:12:14 -08:00
Mosharaf Chowdhury
9915fd22c8
Added BroadcastBT.EndGameFraction config option to control when to start end game.
2010-11-12 15:27:47 -08:00
Mosharaf Chowdhury
c6874d4370
Talk to peers even if current knowledge says they dont have anything to send.
2010-11-12 14:36:34 -08:00
Mosharaf Chowdhury
0adb12e5aa
Merge branch 'mos-bt' into mos-bt-ask-for-block
2010-11-11 14:07:02 -08:00