Timothy Hunter
|
34395730db
|
Someone forgot to pass the parameters: fixes SPARK_MEM set from main script but not passed to executor.
|
2010-12-12 13:30:49 -08:00 |
|
Matei Zaharia
|
0d895ba636
|
Added BSD license
|
2010-12-07 10:32:17 -08:00 |
|
Mosharaf Chowdhury
|
06dc4a5148
|
- Removed config files from git's control.
- Changed DfsShuffle to default in RDD.scala.
|
2010-12-07 10:17:47 -08:00 |
|
Mosharaf Chowdhury
|
f82cc17bc5
|
UseHttpPipelining option is brought back in. It works!
|
2010-12-07 10:07:30 -08:00 |
|
Joshua Hartman
|
799c1b19f5
|
Adding license file for compress-lzf
|
2010-12-07 08:30:29 -08:00 |
|
Joshua Hartman
|
2fb849502f
|
Replacing the native lzf compression code with the ning open-source compress-lzf library. (Apache 2.0 liscense)
|
2010-12-05 21:20:15 -08:00 |
|
Mosharaf Chowdhury
|
7e2d72c328
|
Multiple connections created at a time. No upper limit on the server side though.
|
2010-12-04 18:55:55 -08:00 |
|
Mosharaf Chowdhury
|
c6df327dd7
|
Updated logging format.
|
2010-12-04 16:41:13 -08:00 |
|
Mosharaf Chowdhury
|
7df20d681a
|
Combined MaxRxPeers and MaxTxPeers to a single config parameter MaxConnections
|
2010-12-04 14:37:16 -08:00 |
|
Mosharaf Chowdhury
|
b1745b3103
|
Removed an unnecessary byte array in the middle. Probabaly will have to bring it back if we do block level data movement.
|
2010-12-04 13:55:25 -08:00 |
|
Mosharaf Chowdhury
|
3a671ce989
|
Config parameters are in place. Good to go (I think)
|
2010-12-04 10:59:06 -08:00 |
|
Mosharaf Chowdhury
|
476a216d9d
|
Parallel is working. Need to fix/finalize some config parameters.
|
2010-12-04 02:05:41 -08:00 |
|
Mosharaf Chowdhury
|
c546c299bc
|
combining is happening inside the thread. Its still synchronized though.
|
2010-12-04 00:59:25 -08:00 |
|
Mosharaf Chowdhury
|
0d7ca7751e
|
Bug fixes. Not yet parallel.
|
2010-12-04 00:06:47 -08:00 |
|
Mosharaf Chowdhury
|
52086cef32
|
Building blocks are in place. Still not pulling parallely though.
|
2010-12-03 20:29:39 -08:00 |
|
Mosharaf Chowdhury
|
540a41163f
|
UseHttpPipelining is 'true' by default.
|
2010-12-02 19:56:17 -08:00 |
|
Mosharaf Chowdhury
|
1013a01469
|
Made the root Broadcast object initilization method 'synchronized'.
|
2010-12-02 02:36:21 -08:00 |
|
Mosharaf Chowdhury
|
0de859fbe2
|
Enabling/disabling HTTP pipelining is a config option now. Performance tradeoffs are not obvious yet.
|
2010-12-02 02:32:44 -08:00 |
|
Mosharaf Chowdhury
|
a69638d98d
|
Removed deprecated repl file ClassServer.scala
|
2010-11-30 18:45:04 -08:00 |
|
Mosharaf Chowdhury
|
b7dda4c5bc
|
Merge branch 'multi-tracker' into mos-bt
Conflicts:
conf/java-opts
src/scala/spark/Broadcast.scala
src/scala/spark/DfsBroadcast.scala
src/scala/spark/SparkContext.scala
|
2010-11-30 18:29:38 -08:00 |
|
Mosharaf Chowdhury
|
815ecd349a
|
Made Broadcast Pluggable. Finally!
|
2010-11-30 18:08:49 -08:00 |
|
Mosharaf Chowdhury
|
ea28a30e36
|
Makes transfer faster for small swarms (with size < MaxRxPeers)
|
2010-11-30 16:13:56 -08:00 |
|
Mosharaf Chowdhury
|
191af65fb7
|
- Using the new Cache implementation.
|
2010-11-29 20:35:47 -08:00 |
|
Mosharaf Chowdhury
|
c9cad03c31
|
- Using the new Cache implementation.
- Removed unused code related to dualMode (deprecated).
|
2010-11-29 20:30:38 -08:00 |
|
Mosharaf Chowdhury
|
73714da568
|
Made LocalFileShuffle to be the default.
|
2010-11-29 19:19:21 -08:00 |
|
Mosharaf Chowdhury
|
c62f2fefa5
|
Made LocalFileShuffle to be the default.
|
2010-11-29 19:17:01 -08:00 |
|
Mosharaf Chowdhury
|
15f1f6865d
|
Variable renaming.
|
2010-11-29 14:15:36 -08:00 |
|
Mosharaf Chowdhury
|
bb3e7fbf61
|
Variable renaming.
|
2010-11-29 14:13:35 -08:00 |
|
Mosharaf Chowdhury
|
1157380f21
|
Removed unused third party libraries.
|
2010-11-29 12:21:46 -08:00 |
|
Mosharaf Chowdhury
|
1f7fcb48be
|
Merge branch 'master' into mos-bt
Conflicts:
Makefile
run
src/scala/spark/Broadcast.scala
src/scala/spark/Executor.scala
src/scala/spark/HdfsFile.scala
src/scala/spark/MesosScheduler.scala
src/scala/spark/RDD.scala
src/scala/spark/SparkContext.scala
src/scala/spark/Split.scala
src/scala/spark/Utils.scala
src/scala/spark/repl/SparkInterpreter.scala
third_party/mesos.jar
|
2010-11-29 12:10:06 -08:00 |
|
Mosharaf Chowdhury
|
9e93f8df12
|
Moved broadcast implementations to separate files. Ready to merge with 'master'
|
2010-11-29 11:43:20 -08:00 |
|
Mosharaf Chowdhury
|
8d775448c2
|
- Resolved some of the simpler TODOs related to different timeout and wait periods.
- Removed unused code.
|
2010-11-29 11:20:04 -08:00 |
|
Mosharaf Chowdhury
|
8494b3a4f9
|
- Added log messages for benchmarking.
- Added GroupByTest.scala for benchmarking.
|
2010-11-27 23:51:43 -08:00 |
|
Mosharaf Chowdhury
|
19a6b194a9
|
Merge branch 'master' into multi-tracker
Conflicts:
Makefile
run
src/scala/spark/Broadcast.scala
src/scala/spark/Executor.scala
src/scala/spark/HdfsFile.scala
src/scala/spark/MesosScheduler.scala
src/scala/spark/RDD.scala
src/scala/spark/SparkContext.scala
src/scala/spark/Split.scala
src/scala/spark/Utils.scala
src/scala/spark/repl/SparkInterpreter.scala
third_party/mesos.jar
|
2010-11-27 01:27:20 -08:00 |
|
Mosharaf Chowdhury
|
e4b8db45ae
|
- Moved DaemonThreadPool factory methods to the Broadcast Object.
- Reflection-based broadcast class loading is still not working.
|
2010-11-27 00:53:57 -08:00 |
|
Mosharaf Chowdhury
|
c25914228c
|
Moved ChaninedStreaming to a separate file and renamed to ChainedBroadcast.
|
2010-11-26 23:15:33 -08:00 |
|
Mosharaf Chowdhury
|
98542f81bb
|
- Removed dual Mode
- Removed ununsed speed-related stuff
- Moved (H)DfsBroadcast to its own file
|
2010-11-26 22:58:35 -08:00 |
|
Mosharaf Chowdhury
|
9d0111659b
|
Solved compilation issue. Should've been more careful to do a clean compilation before committing.
|
2010-11-26 22:10:38 -08:00 |
|
Mosharaf Chowdhury
|
9c8a65b920
|
Removed everything related to SplitStreaming...
|
2010-11-26 22:06:43 -08:00 |
|
Mosharaf Chowdhury
|
830496b901
|
Updated some comments.
|
2010-11-25 23:49:57 -08:00 |
|
Mosharaf Chowdhury
|
690b55917d
|
Slaves weren't printing hostAddress. Fixed.
|
2010-11-18 10:53:34 -08:00 |
|
Mosharaf Chowdhury
|
2de1664214
|
Slaves weren't logging hostAddress. Fixed.
|
2010-11-18 10:51:57 -08:00 |
|
Mosharaf Chowdhury
|
dfa6540fdd
|
Everyone logs local hostAddress.
|
2010-11-18 09:58:46 -08:00 |
|
Mosharaf Chowdhury
|
1f2662d45a
|
Everyone logs their local hostAddress.
|
2010-11-18 09:55:19 -08:00 |
|
Matei Zaharia
|
dcfa2ce83b
|
Further improvements -- build native stuff in target directory and add a
test-report target for XML test reports
|
2010-11-14 00:46:19 -08:00 |
|
Matei Zaharia
|
e86b620f9e
|
Fixed some more stuff (Eclispe target and native build)
|
2010-11-13 22:46:00 -08:00 |
|
Matei Zaharia
|
89fcd96702
|
Initial work to get Spark compiling with SBT 0.7.5 RC0
|
2010-11-13 22:07:08 -08:00 |
|
Mosharaf Chowdhury
|
c3f354c6fc
|
Guide now select random peers to select sources instead of rolling statically.
Random is good. Deterministic is bad. Well, mostly.
|
2010-11-13 20:18:48 -08:00 |
|
Matei Zaharia
|
b7574201d5
|
Added some stuff to .gitignore
|
2010-11-13 18:41:35 -08:00 |
|
Matei Zaharia
|
f8ea98d989
|
Remove -unchecked compiler parameter
|
2010-11-13 18:39:07 -08:00 |
|