Mosharaf Chowdhury
a83a722256
Renamed BlockedLocalFileShuffle to HttpBlockedLocalFileShuffle for merging with the mos-shuffle branch.
2010-12-19 14:02:19 -08:00
Mosharaf Chowdhury
62d61ed928
- Reimplemented BlockedLocalFileShuffle without creating too many files.
...
- Clients now request for byte ranges to the server using an INDEX file.
2010-12-18 14:03:49 -08:00
Mosharaf Chowdhury
5c5d767bc1
Modified MultiBroadcastTest.
2010-12-18 10:40:00 -08:00
Mosharaf Chowdhury
d18d08ec9d
Added a new BroadcastTest in the examples where 2 broadcasts are required. Should be used to experiment how multiple broadcasts work.
2010-12-17 10:43:49 -08:00
Mosharaf Chowdhury
e30fdeb025
Updated GroupByKey example.
2010-12-16 20:30:18 -08:00
Mosharaf Chowdhury
a40cbc1904
Code formatting.
2010-12-16 16:54:02 -08:00
Mosharaf Chowdhury
ce96d8a7d3
First version of BlockedLocalFileShuffle is in. It works!
2010-12-16 15:15:51 -08:00
Mosharaf Chowdhury
fddcdf87c9
Added a small description of how ParallelLFS works.
2010-12-16 11:58:00 -08:00
Mosharaf Chowdhury
77a4017585
Fixed config param naming in ParallelLocalFileShuffle
2010-12-16 11:42:37 -08:00
Mosharaf Chowdhury
c5483e39f9
- ParallelLocalFileShuffle does NOT use HttpPipelining at all.
...
- Config option related to pipelining has been removed.
- Summary: Basic -> Pipelining / Parallel -> NO pipelining
2010-12-15 22:08:34 -08:00
Mosharaf Chowdhury
56d8a2afa1
- Updated java-opts file of this branch.
...
- Renamed some ParallelLocalFileShuffle config options for clarity.
2010-12-15 20:56:22 -08:00
Mosharaf Chowdhury
25fb3c4cf6
- Brought back Matei's LocalFileShuffle implementation as BasicLocalFileShuffle
...
- Renamed parallel-pull version to ParallelLocalFileShuffle
- Note that setting max-concurrent connections to 1 in ParallelLocalFileShuffle should essentially be the same as BasicLocalFileShuffle
2010-12-15 20:33:28 -08:00
Matei Zaharia
817e722321
Merge branch 'master' of github.com:mesos/spark
2010-12-15 19:40:35 -08:00
Matei Zaharia
14c29c1b14
Fixed import
2010-12-15 19:40:27 -08:00
Mosharaf Chowdhury
5cafdd7ba2
Removed some unused imports from Broadcast.scala
2010-12-15 19:11:23 -08:00
Mosharaf Chowdhury
be0ce57de2
- Fixed an compilation error due to wrong 'import' of legacy lzf libraries in DfsBroadcast.scala
...
- Updated to use ning libraries.
- Passes all unit tests
2010-12-15 18:34:27 -08:00
Matei Zaharia
5c222dbe28
Merge branch 'master' into mos-bt
...
Conflicts:
src/scala/spark/Broadcast.scala
2010-12-15 10:57:39 -08:00
Mosharaf Chowdhury
0a5c24ae3d
- Default broadcast mechanism is set to DfsBroadcast
...
- Configuration parameters are renamed to follow our convention
- Master now automatically supplies its hostAddress instead of reading from config file
- sendBroadcast has been removed from the Broadcast trait
2010-12-13 14:36:39 -08:00
Timothy Hunter
34395730db
Someone forgot to pass the parameters: fixes SPARK_MEM set from main script but not passed to executor.
2010-12-12 13:30:49 -08:00
Matei Zaharia
0d895ba636
Added BSD license
2010-12-07 10:32:17 -08:00
Mosharaf Chowdhury
06dc4a5148
- Removed config files from git's control.
...
- Changed DfsShuffle to default in RDD.scala.
2010-12-07 10:17:47 -08:00
Mosharaf Chowdhury
f82cc17bc5
UseHttpPipelining option is brought back in. It works!
2010-12-07 10:07:30 -08:00
Joshua Hartman
799c1b19f5
Adding license file for compress-lzf
2010-12-07 08:30:29 -08:00
Joshua Hartman
2fb849502f
Replacing the native lzf compression code with the ning open-source compress-lzf library. (Apache 2.0 liscense)
2010-12-05 21:20:15 -08:00
Mosharaf Chowdhury
7e2d72c328
Multiple connections created at a time. No upper limit on the server side though.
2010-12-04 18:55:55 -08:00
Mosharaf Chowdhury
c6df327dd7
Updated logging format.
2010-12-04 16:41:13 -08:00
Mosharaf Chowdhury
7df20d681a
Combined MaxRxPeers and MaxTxPeers to a single config parameter MaxConnections
2010-12-04 14:37:16 -08:00
Mosharaf Chowdhury
b1745b3103
Removed an unnecessary byte array in the middle. Probabaly will have to bring it back if we do block level data movement.
2010-12-04 13:55:25 -08:00
Mosharaf Chowdhury
3a671ce989
Config parameters are in place. Good to go (I think)
2010-12-04 10:59:06 -08:00
Mosharaf Chowdhury
476a216d9d
Parallel is working. Need to fix/finalize some config parameters.
2010-12-04 02:05:41 -08:00
Mosharaf Chowdhury
c546c299bc
combining is happening inside the thread. Its still synchronized though.
2010-12-04 00:59:25 -08:00
Mosharaf Chowdhury
0d7ca7751e
Bug fixes. Not yet parallel.
2010-12-04 00:06:47 -08:00
Mosharaf Chowdhury
52086cef32
Building blocks are in place. Still not pulling parallely though.
2010-12-03 20:29:39 -08:00
Mosharaf Chowdhury
540a41163f
UseHttpPipelining is 'true' by default.
2010-12-02 19:56:17 -08:00
Mosharaf Chowdhury
1013a01469
Made the root Broadcast object initilization method 'synchronized'.
2010-12-02 02:36:21 -08:00
Mosharaf Chowdhury
0de859fbe2
Enabling/disabling HTTP pipelining is a config option now. Performance tradeoffs are not obvious yet.
2010-12-02 02:32:44 -08:00
Mosharaf Chowdhury
a69638d98d
Removed deprecated repl file ClassServer.scala
2010-11-30 18:45:04 -08:00
Mosharaf Chowdhury
b7dda4c5bc
Merge branch 'multi-tracker' into mos-bt
...
Conflicts:
conf/java-opts
src/scala/spark/Broadcast.scala
src/scala/spark/DfsBroadcast.scala
src/scala/spark/SparkContext.scala
2010-11-30 18:29:38 -08:00
Mosharaf Chowdhury
815ecd349a
Made Broadcast Pluggable. Finally!
2010-11-30 18:08:49 -08:00
Mosharaf Chowdhury
ea28a30e36
Makes transfer faster for small swarms (with size < MaxRxPeers)
2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury
191af65fb7
- Using the new Cache implementation.
2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury
c9cad03c31
- Using the new Cache implementation.
...
- Removed unused code related to dualMode (deprecated).
2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury
73714da568
Made LocalFileShuffle to be the default.
2010-11-29 19:19:21 -08:00
Mosharaf Chowdhury
c62f2fefa5
Made LocalFileShuffle to be the default.
2010-11-29 19:17:01 -08:00
Mosharaf Chowdhury
15f1f6865d
Variable renaming.
2010-11-29 14:15:36 -08:00
Mosharaf Chowdhury
bb3e7fbf61
Variable renaming.
2010-11-29 14:13:35 -08:00
Mosharaf Chowdhury
1157380f21
Removed unused third party libraries.
2010-11-29 12:21:46 -08:00
Mosharaf Chowdhury
1f7fcb48be
Merge branch 'master' into mos-bt
...
Conflicts:
Makefile
run
src/scala/spark/Broadcast.scala
src/scala/spark/Executor.scala
src/scala/spark/HdfsFile.scala
src/scala/spark/MesosScheduler.scala
src/scala/spark/RDD.scala
src/scala/spark/SparkContext.scala
src/scala/spark/Split.scala
src/scala/spark/Utils.scala
src/scala/spark/repl/SparkInterpreter.scala
third_party/mesos.jar
2010-11-29 12:10:06 -08:00
Mosharaf Chowdhury
9e93f8df12
Moved broadcast implementations to separate files. Ready to merge with 'master'
2010-11-29 11:43:20 -08:00
Mosharaf Chowdhury
8d775448c2
- Resolved some of the simpler TODOs related to different timeout and wait periods.
...
- Removed unused code.
2010-11-29 11:20:04 -08:00