ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
240 commits 6 branches 181 tags 397 MiB
Commit graph

240 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury ba74169162 Dummy log line added in SparkContext for #45 2011-01-06 21:21:35 -08:00
Mosharaf Chowdhury e76b4f7155 Turned OFF some log messages due to #45. 2011-01-06 17:07:14 -08:00
Mosharaf Chowdhury f5ba0b49c6 Bug fix + default config param values update. 
Updated some default config parameters for broadcast parameters.
Fixed a concurrency bug in the variable that keeps track of how many receivers have finished.
 2011-01-06 15:44:24 -08:00
Mosharaf Chowdhury 2fefbe17e4 TreeBroadcast is an extended version of ChainedBroadcast with customizable maxDegree per node. maxDegree = 1 is ChainedBroadcast. 2010-12-28 20:18:49 -08:00
Mosharaf Chowdhury 5540a99ab7 ChainedBroadcast is also reading masterHostAddress from config file until #42 is resolved. 2010-12-28 18:53:59 -08:00
Mosharaf Chowdhury 20eec59f04 Bug fix + formatting. 2010-12-25 22:54:46 -08:00
Mosharaf Chowdhury ba71b61e40 Reading masterHostAddress from config file until #42 has been resolved. 2010-12-25 10:22:22 -08:00
Mosharaf Chowdhury 5c5d767bc1 Modified MultiBroadcastTest. 2010-12-18 10:40:00 -08:00
Mosharaf Chowdhury d18d08ec9d Added a new BroadcastTest in the examples where 2 broadcasts are required. Should be used to experiment how multiple broadcasts work. 2010-12-17 10:43:49 -08:00
Mosharaf Chowdhury 5cafdd7ba2 Removed some unused imports from Broadcast.scala 2010-12-15 19:11:23 -08:00
Mosharaf Chowdhury be0ce57de2 - Fixed an compilation error due to wrong 'import' of legacy lzf libraries in DfsBroadcast.scala 
- Updated to use ning libraries.
 - Passes all unit tests
 2010-12-15 18:34:27 -08:00
Matei Zaharia 5c222dbe28 Merge branch 'master' into mos-bt 
Conflicts:
	src/scala/spark/Broadcast.scala
 2010-12-15 10:57:39 -08:00
Mosharaf Chowdhury 0a5c24ae3d - Default broadcast mechanism is set to DfsBroadcast 
- Configuration parameters are renamed to follow our convention
 - Master now automatically supplies its hostAddress instead of reading from config file
 - sendBroadcast has been removed from the Broadcast trait
 2010-12-13 14:36:39 -08:00
Timothy Hunter 34395730db Someone forgot to pass the parameters: fixes SPARK_MEM set from main script but not passed to executor. 2010-12-12 13:30:49 -08:00
Matei Zaharia 0d895ba636 Added BSD license 2010-12-07 10:32:17 -08:00
Mosharaf Chowdhury 06dc4a5148 - Removed config files from git's control. 
- Changed DfsShuffle to default in RDD.scala.
 2010-12-07 10:17:47 -08:00
Joshua Hartman 799c1b19f5 Adding license file for compress-lzf 2010-12-07 08:30:29 -08:00
Joshua Hartman 2fb849502f Replacing the native lzf compression code with the ning open-source compress-lzf library. (Apache 2.0 liscense) 2010-12-05 21:20:15 -08:00
Mosharaf Chowdhury 1013a01469 Made the root Broadcast object initilization method 'synchronized'. 2010-12-02 02:36:21 -08:00
Mosharaf Chowdhury a69638d98d Removed deprecated repl file ClassServer.scala 2010-11-30 18:45:04 -08:00
Mosharaf Chowdhury b7dda4c5bc Merge branch 'multi-tracker' into mos-bt 
Conflicts:
	conf/java-opts
	src/scala/spark/Broadcast.scala
	src/scala/spark/DfsBroadcast.scala
	src/scala/spark/SparkContext.scala
 2010-11-30 18:29:38 -08:00
Mosharaf Chowdhury 815ecd349a Made Broadcast Pluggable. Finally! 2010-11-30 18:08:49 -08:00
Mosharaf Chowdhury ea28a30e36 Makes transfer faster for small swarms (with size < MaxRxPeers) 2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury 191af65fb7 - Using the new Cache implementation. 2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury c9cad03c31 - Using the new Cache implementation. 
- Removed unused code related to dualMode (deprecated).
 2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury 73714da568 Made LocalFileShuffle to be the default. 2010-11-29 19:19:21 -08:00
Mosharaf Chowdhury c62f2fefa5 Made LocalFileShuffle to be the default. 2010-11-29 19:17:01 -08:00
Mosharaf Chowdhury 15f1f6865d Variable renaming. 2010-11-29 14:15:36 -08:00
Mosharaf Chowdhury bb3e7fbf61 Variable renaming. 2010-11-29 14:13:35 -08:00
Mosharaf Chowdhury 1157380f21 Removed unused third party libraries. 2010-11-29 12:21:46 -08:00
Mosharaf Chowdhury 1f7fcb48be Merge branch 'master' into mos-bt 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/Executor.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/MesosScheduler.scala
	src/scala/spark/RDD.scala
	src/scala/spark/SparkContext.scala
	src/scala/spark/Split.scala
	src/scala/spark/Utils.scala
	src/scala/spark/repl/SparkInterpreter.scala
	third_party/mesos.jar
 2010-11-29 12:10:06 -08:00
Mosharaf Chowdhury 9e93f8df12 Moved broadcast implementations to separate files. Ready to merge with 'master' 2010-11-29 11:43:20 -08:00
Mosharaf Chowdhury 8d775448c2 - Resolved some of the simpler TODOs related to different timeout and wait periods. 
- Removed unused code.
 2010-11-29 11:20:04 -08:00
Mosharaf Chowdhury 19a6b194a9 Merge branch 'master' into multi-tracker 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/Executor.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/MesosScheduler.scala
	src/scala/spark/RDD.scala
	src/scala/spark/SparkContext.scala
	src/scala/spark/Split.scala
	src/scala/spark/Utils.scala
	src/scala/spark/repl/SparkInterpreter.scala
	third_party/mesos.jar
 2010-11-27 01:27:20 -08:00
Mosharaf Chowdhury e4b8db45ae - Moved DaemonThreadPool factory methods to the Broadcast Object. 
- Reflection-based broadcast class loading is still not working.
 2010-11-27 00:53:57 -08:00
Mosharaf Chowdhury c25914228c Moved ChaninedStreaming to a separate file and renamed to ChainedBroadcast. 2010-11-26 23:15:33 -08:00
Mosharaf Chowdhury 98542f81bb - Removed dual Mode 
- Removed ununsed speed-related stuff
- Moved (H)DfsBroadcast to its own file
 2010-11-26 22:58:35 -08:00
Mosharaf Chowdhury 9d0111659b Solved compilation issue. Should've been more careful to do a clean compilation before committing. 2010-11-26 22:10:38 -08:00
Mosharaf Chowdhury 9c8a65b920 Removed everything related to SplitStreaming... 2010-11-26 22:06:43 -08:00
Mosharaf Chowdhury 830496b901 Updated some comments. 2010-11-25 23:49:57 -08:00
Mosharaf Chowdhury 690b55917d Slaves weren't printing hostAddress. Fixed. 2010-11-18 10:53:34 -08:00
Mosharaf Chowdhury 2de1664214 Slaves weren't logging hostAddress. Fixed. 2010-11-18 10:51:57 -08:00
Mosharaf Chowdhury dfa6540fdd Everyone logs local hostAddress. 2010-11-18 09:58:46 -08:00
Mosharaf Chowdhury 1f2662d45a Everyone logs their local hostAddress. 2010-11-18 09:55:19 -08:00
Mosharaf Chowdhury c3f354c6fc Guide now select random peers to select sources instead of rolling statically. 
Random is good. Deterministic is bad. Well, mostly.
 2010-11-13 20:18:48 -08:00
Matei Zaharia f8ea98d989 Remove -unchecked compiler parameter 2010-11-13 18:39:07 -08:00
Matei Zaharia f8966ffc11 Added a shuffle test with negative hash codes for some keys (this was a bug earlier) 2010-11-12 16:18:45 -08:00
Matei Zaharia d0a9966555 Unit tests for shuffle operations. Fixes #33. 2010-11-12 16:12:14 -08:00
Mosharaf Chowdhury 9915fd22c8 Added BroadcastBT.EndGameFraction config option to control when to start end game. 2010-11-12 15:27:47 -08:00
Mosharaf Chowdhury c6874d4370 Talk to peers even if current knowledge says they dont have anything to send. 2010-11-12 14:36:34 -08:00