Mosharaf Chowdhury
1013a01469
Made the root Broadcast object initilization method 'synchronized'.
2010-12-02 02:36:21 -08:00
Mosharaf Chowdhury
0de859fbe2
Enabling/disabling HTTP pipelining is a config option now. Performance tradeoffs are not obvious yet.
2010-12-02 02:32:44 -08:00
Mosharaf Chowdhury
a69638d98d
Removed deprecated repl file ClassServer.scala
2010-11-30 18:45:04 -08:00
Mosharaf Chowdhury
b7dda4c5bc
Merge branch 'multi-tracker' into mos-bt
...
Conflicts:
conf/java-opts
src/scala/spark/Broadcast.scala
src/scala/spark/DfsBroadcast.scala
src/scala/spark/SparkContext.scala
2010-11-30 18:29:38 -08:00
Mosharaf Chowdhury
815ecd349a
Made Broadcast Pluggable. Finally!
2010-11-30 18:08:49 -08:00
Mosharaf Chowdhury
ea28a30e36
Makes transfer faster for small swarms (with size < MaxRxPeers)
2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury
191af65fb7
- Using the new Cache implementation.
2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury
c9cad03c31
- Using the new Cache implementation.
...
- Removed unused code related to dualMode (deprecated).
2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury
73714da568
Made LocalFileShuffle to be the default.
2010-11-29 19:19:21 -08:00
Mosharaf Chowdhury
c62f2fefa5
Made LocalFileShuffle to be the default.
2010-11-29 19:17:01 -08:00
Mosharaf Chowdhury
15f1f6865d
Variable renaming.
2010-11-29 14:15:36 -08:00
Mosharaf Chowdhury
bb3e7fbf61
Variable renaming.
2010-11-29 14:13:35 -08:00
Mosharaf Chowdhury
1157380f21
Removed unused third party libraries.
2010-11-29 12:21:46 -08:00
Mosharaf Chowdhury
1f7fcb48be
Merge branch 'master' into mos-bt
...
Conflicts:
Makefile
run
src/scala/spark/Broadcast.scala
src/scala/spark/Executor.scala
src/scala/spark/HdfsFile.scala
src/scala/spark/MesosScheduler.scala
src/scala/spark/RDD.scala
src/scala/spark/SparkContext.scala
src/scala/spark/Split.scala
src/scala/spark/Utils.scala
src/scala/spark/repl/SparkInterpreter.scala
third_party/mesos.jar
2010-11-29 12:10:06 -08:00
Mosharaf Chowdhury
9e93f8df12
Moved broadcast implementations to separate files. Ready to merge with 'master'
2010-11-29 11:43:20 -08:00
Mosharaf Chowdhury
8d775448c2
- Resolved some of the simpler TODOs related to different timeout and wait periods.
...
- Removed unused code.
2010-11-29 11:20:04 -08:00
Mosharaf Chowdhury
8494b3a4f9
- Added log messages for benchmarking.
...
- Added GroupByTest.scala for benchmarking.
2010-11-27 23:51:43 -08:00
Mosharaf Chowdhury
19a6b194a9
Merge branch 'master' into multi-tracker
...
Conflicts:
Makefile
run
src/scala/spark/Broadcast.scala
src/scala/spark/Executor.scala
src/scala/spark/HdfsFile.scala
src/scala/spark/MesosScheduler.scala
src/scala/spark/RDD.scala
src/scala/spark/SparkContext.scala
src/scala/spark/Split.scala
src/scala/spark/Utils.scala
src/scala/spark/repl/SparkInterpreter.scala
third_party/mesos.jar
2010-11-27 01:27:20 -08:00
Mosharaf Chowdhury
e4b8db45ae
- Moved DaemonThreadPool factory methods to the Broadcast Object.
...
- Reflection-based broadcast class loading is still not working.
2010-11-27 00:53:57 -08:00
Mosharaf Chowdhury
c25914228c
Moved ChaninedStreaming to a separate file and renamed to ChainedBroadcast.
2010-11-26 23:15:33 -08:00
Mosharaf Chowdhury
98542f81bb
- Removed dual Mode
...
- Removed ununsed speed-related stuff
- Moved (H)DfsBroadcast to its own file
2010-11-26 22:58:35 -08:00
Mosharaf Chowdhury
9d0111659b
Solved compilation issue. Should've been more careful to do a clean compilation before committing.
2010-11-26 22:10:38 -08:00
Mosharaf Chowdhury
9c8a65b920
Removed everything related to SplitStreaming...
2010-11-26 22:06:43 -08:00
Mosharaf Chowdhury
830496b901
Updated some comments.
2010-11-25 23:49:57 -08:00
Mosharaf Chowdhury
690b55917d
Slaves weren't printing hostAddress. Fixed.
2010-11-18 10:53:34 -08:00
Mosharaf Chowdhury
2de1664214
Slaves weren't logging hostAddress. Fixed.
2010-11-18 10:51:57 -08:00
Mosharaf Chowdhury
dfa6540fdd
Everyone logs local hostAddress.
2010-11-18 09:58:46 -08:00
Mosharaf Chowdhury
1f2662d45a
Everyone logs their local hostAddress.
2010-11-18 09:55:19 -08:00
Matei Zaharia
dcfa2ce83b
Further improvements -- build native stuff in target directory and add a
...
test-report target for XML test reports
2010-11-14 00:46:19 -08:00
Matei Zaharia
e86b620f9e
Fixed some more stuff (Eclispe target and native build)
2010-11-13 22:46:00 -08:00
Matei Zaharia
89fcd96702
Initial work to get Spark compiling with SBT 0.7.5 RC0
2010-11-13 22:07:08 -08:00
Mosharaf Chowdhury
c3f354c6fc
Guide now select random peers to select sources instead of rolling statically.
...
Random is good. Deterministic is bad. Well, mostly.
2010-11-13 20:18:48 -08:00
Matei Zaharia
b7574201d5
Added some stuff to .gitignore
2010-11-13 18:41:35 -08:00
Matei Zaharia
f8ea98d989
Remove -unchecked compiler parameter
2010-11-13 18:39:07 -08:00
Matei Zaharia
af74217c96
Undo JLine fix that turns out to only be needed when buildr is running
...
on JRuby. This is quite ugly: JRuby has its own version of JLine which
is older than Scala's, and JLine changed API in such a way that code
written for the new version won't compile with the old one and vice
versa. Sadly, this might be a reason to drop buildr, unless we can
package a JRuby with it that uses the right version, or we ask people to
only use the C Ruby version of buildr (which doesn't work on OS X right
now!)
2010-11-13 18:38:50 -08:00
Matei Zaharia
b84769a107
Modified project structure to work with buildr
2010-11-13 17:18:05 -08:00
Matei Zaharia
f8966ffc11
Added a shuffle test with negative hash codes for some keys (this was a bug earlier)
2010-11-12 16:18:45 -08:00
Matei Zaharia
d0a9966555
Unit tests for shuffle operations. Fixes #33 .
2010-11-12 16:12:14 -08:00
Mosharaf Chowdhury
9915fd22c8
Added BroadcastBT.EndGameFraction config option to control when to start end game.
2010-11-12 15:27:47 -08:00
Mosharaf Chowdhury
c6874d4370
Talk to peers even if current knowledge says they dont have anything to send.
2010-11-12 14:36:34 -08:00
Mosharaf Chowdhury
0adb12e5aa
Merge branch 'mos-bt' into mos-bt-ask-for-block
2010-11-11 14:07:02 -08:00
Mosharaf Chowdhury
23ebac8c30
This reverts the addToListOfSources methods of commit 642098c211
.
2010-11-11 14:05:38 -08:00
Mosharaf Chowdhury
88d08b7a89
Bug fix.
2010-11-10 17:04:14 -08:00
Mosharaf Chowdhury
eb70f6a36e
Receivers explicitly ask for blocks to send.
...
The Random generator must be updated so that everyone does not ask for the same block during the initial phase.
2010-11-10 16:47:45 -08:00
Mosharaf Chowdhury
d7579e743a
Style update: all functions that return void should have a Unit return type.
2010-11-10 10:32:29 -08:00
Mosharaf Chowdhury
642098c211
Bug fixes and/or minor optimization.
2010-11-09 23:59:06 -08:00
Mosharaf Chowdhury
b58bbadbfb
All daemon => Fast shutdown :)
2010-11-09 17:05:19 -08:00
Mosharaf Chowdhury
76610e842b
Removed some unnecessary stuff.
2010-11-09 16:59:32 -08:00
Mosharaf Chowdhury
50612ab3a9
Every thread is a daemon thread => Program shuts down ASAP once its done.
...
Some log messages added to blank exception handlers.
2010-11-09 16:44:42 -08:00
Matei Zaharia
7b25ab87af
Added options for using an external HTTP server with LocalFileShuffle
2010-11-09 13:46:30 -08:00