ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
21813 commits 6 branches 181 tags 397 MiB
Commit graph

21813 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury 62d61ed928 - Reimplemented BlockedLocalFileShuffle without creating too many files. 
- Clients now request for byte ranges to the server using an INDEX file.
 2010-12-18 14:03:49 -08:00
Mosharaf Chowdhury 5c5d767bc1 Modified MultiBroadcastTest. 2010-12-18 10:40:00 -08:00
Mosharaf Chowdhury d18d08ec9d Added a new BroadcastTest in the examples where 2 broadcasts are required. Should be used to experiment how multiple broadcasts work. 2010-12-17 10:43:49 -08:00
Mosharaf Chowdhury e30fdeb025 Updated GroupByKey example. 2010-12-16 20:30:18 -08:00
Mosharaf Chowdhury a40cbc1904 Code formatting. 2010-12-16 16:54:02 -08:00
Mosharaf Chowdhury ce96d8a7d3 First version of BlockedLocalFileShuffle is in. It works! 2010-12-16 15:15:51 -08:00
Mosharaf Chowdhury fddcdf87c9 Added a small description of how ParallelLFS works. 2010-12-16 11:58:00 -08:00
Mosharaf Chowdhury 77a4017585 Fixed config param naming in ParallelLocalFileShuffle 2010-12-16 11:42:37 -08:00
Mosharaf Chowdhury c5483e39f9 - ParallelLocalFileShuffle does NOT use HttpPipelining at all. 
- Config option related to pipelining has been removed.
 - Summary: Basic -> Pipelining / Parallel -> NO pipelining
 2010-12-15 22:08:34 -08:00
Mosharaf Chowdhury 56d8a2afa1 - Updated java-opts file of this branch. 
- Renamed some ParallelLocalFileShuffle config options for clarity.
 2010-12-15 20:56:22 -08:00
Mosharaf Chowdhury 25fb3c4cf6 - Brought back Matei's LocalFileShuffle implementation as BasicLocalFileShuffle 
- Renamed parallel-pull version to ParallelLocalFileShuffle
 - Note that setting max-concurrent connections to 1 in ParallelLocalFileShuffle should essentially be the same as BasicLocalFileShuffle
 2010-12-15 20:33:28 -08:00
Matei Zaharia 817e722321 Merge branch 'master' of github.com:mesos/spark 2010-12-15 19:40:35 -08:00
Matei Zaharia 14c29c1b14 Fixed import 2010-12-15 19:40:27 -08:00
Mosharaf Chowdhury 5cafdd7ba2 Removed some unused imports from Broadcast.scala 2010-12-15 19:11:23 -08:00
Mosharaf Chowdhury be0ce57de2 - Fixed an compilation error due to wrong 'import' of legacy lzf libraries in DfsBroadcast.scala 
- Updated to use ning libraries.
 - Passes all unit tests
 2010-12-15 18:34:27 -08:00
Matei Zaharia 5c222dbe28 Merge branch 'master' into mos-bt 
Conflicts:
	src/scala/spark/Broadcast.scala
 2010-12-15 10:57:39 -08:00
Mosharaf Chowdhury 0a5c24ae3d - Default broadcast mechanism is set to DfsBroadcast 
- Configuration parameters are renamed to follow our convention
 - Master now automatically supplies its hostAddress instead of reading from config file
 - sendBroadcast has been removed from the Broadcast trait
 2010-12-13 14:36:39 -08:00
Timothy Hunter 34395730db Someone forgot to pass the parameters: fixes SPARK_MEM set from main script but not passed to executor. 2010-12-12 13:30:49 -08:00
Matei Zaharia 0d895ba636 Added BSD license 2010-12-07 10:32:17 -08:00
Mosharaf Chowdhury 06dc4a5148 - Removed config files from git's control. 
- Changed DfsShuffle to default in RDD.scala.
 2010-12-07 10:17:47 -08:00
Mosharaf Chowdhury f82cc17bc5 UseHttpPipelining option is brought back in. It works! 2010-12-07 10:07:30 -08:00
Joshua Hartman 799c1b19f5 Adding license file for compress-lzf 2010-12-07 08:30:29 -08:00
Joshua Hartman 2fb849502f Replacing the native lzf compression code with the ning open-source compress-lzf library. (Apache 2.0 liscense) 2010-12-05 21:20:15 -08:00
Mosharaf Chowdhury 7e2d72c328 Multiple connections created at a time. No upper limit on the server side though. 2010-12-04 18:55:55 -08:00
Mosharaf Chowdhury c6df327dd7 Updated logging format. 2010-12-04 16:41:13 -08:00
Mosharaf Chowdhury 7df20d681a Combined MaxRxPeers and MaxTxPeers to a single config parameter MaxConnections 2010-12-04 14:37:16 -08:00
Mosharaf Chowdhury b1745b3103 Removed an unnecessary byte array in the middle. Probabaly will have to bring it back if we do block level data movement. 2010-12-04 13:55:25 -08:00
Mosharaf Chowdhury 3a671ce989 Config parameters are in place. Good to go (I think) 2010-12-04 10:59:06 -08:00
Mosharaf Chowdhury 476a216d9d Parallel is working. Need to fix/finalize some config parameters. 2010-12-04 02:05:41 -08:00
Mosharaf Chowdhury c546c299bc combining is happening inside the thread. Its still synchronized though. 2010-12-04 00:59:25 -08:00
Mosharaf Chowdhury 0d7ca7751e Bug fixes. Not yet parallel. 2010-12-04 00:06:47 -08:00
Mosharaf Chowdhury 52086cef32 Building blocks are in place. Still not pulling parallely though. 2010-12-03 20:29:39 -08:00
Mosharaf Chowdhury 540a41163f UseHttpPipelining is 'true' by default. 2010-12-02 19:56:17 -08:00
Mosharaf Chowdhury 1013a01469 Made the root Broadcast object initilization method 'synchronized'. 2010-12-02 02:36:21 -08:00
Mosharaf Chowdhury 0de859fbe2 Enabling/disabling HTTP pipelining is a config option now. Performance tradeoffs are not obvious yet. 2010-12-02 02:32:44 -08:00
Mosharaf Chowdhury a69638d98d Removed deprecated repl file ClassServer.scala 2010-11-30 18:45:04 -08:00
Mosharaf Chowdhury b7dda4c5bc Merge branch 'multi-tracker' into mos-bt 
Conflicts:
	conf/java-opts
	src/scala/spark/Broadcast.scala
	src/scala/spark/DfsBroadcast.scala
	src/scala/spark/SparkContext.scala
 2010-11-30 18:29:38 -08:00
Mosharaf Chowdhury 815ecd349a Made Broadcast Pluggable. Finally! 2010-11-30 18:08:49 -08:00
Mosharaf Chowdhury ea28a30e36 Makes transfer faster for small swarms (with size < MaxRxPeers) 2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury 191af65fb7 - Using the new Cache implementation. 2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury c9cad03c31 - Using the new Cache implementation. 
- Removed unused code related to dualMode (deprecated).
 2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury 73714da568 Made LocalFileShuffle to be the default. 2010-11-29 19:19:21 -08:00
Mosharaf Chowdhury c62f2fefa5 Made LocalFileShuffle to be the default. 2010-11-29 19:17:01 -08:00
Mosharaf Chowdhury 15f1f6865d Variable renaming. 2010-11-29 14:15:36 -08:00
Mosharaf Chowdhury bb3e7fbf61 Variable renaming. 2010-11-29 14:13:35 -08:00
Mosharaf Chowdhury 1157380f21 Removed unused third party libraries. 2010-11-29 12:21:46 -08:00
Mosharaf Chowdhury 1f7fcb48be Merge branch 'master' into mos-bt 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/Executor.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/MesosScheduler.scala
	src/scala/spark/RDD.scala
	src/scala/spark/SparkContext.scala
	src/scala/spark/Split.scala
	src/scala/spark/Utils.scala
	src/scala/spark/repl/SparkInterpreter.scala
	third_party/mesos.jar
 2010-11-29 12:10:06 -08:00
Mosharaf Chowdhury 9e93f8df12 Moved broadcast implementations to separate files. Ready to merge with 'master' 2010-11-29 11:43:20 -08:00
Mosharaf Chowdhury 8d775448c2 - Resolved some of the simpler TODOs related to different timeout and wait periods. 
- Removed unused code.
 2010-11-29 11:20:04 -08:00
Mosharaf Chowdhury 8494b3a4f9 - Added log messages for benchmarking. 
- Added GroupByTest.scala for benchmarking.
 2010-11-27 23:51:43 -08:00