ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
270 commits 6 branches 181 tags 397 MiB
Commit graph

270 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury 520bbdc7e3 Peers now gossip about their neighbors when they talk. 2011-02-10 20:15:30 -08:00
Mosharaf Chowdhury 8034728afc Merge branch 'master' into mos-bt 2011-02-09 15:30:41 -08:00
Mosharaf Chowdhury 441462bc7f Fixed some warnings during compilation. 2011-02-09 12:11:43 -08:00
Matei Zaharia 62f1c6f5a8 Remove build.properties from version control 2011-02-09 11:52:56 -08:00
Mosharaf Chowdhury 1a73c0d265 Merged with master. Using sbt. 2011-02-09 10:48:48 -08:00
Mosharaf Chowdhury 495b38658e Merge branch 'master' into mos-bt 2011-02-09 10:40:23 -08:00
Matei Zaharia d3df963a13 Brought in some reorganization of build file from Hive branch 2011-02-08 21:27:36 -08:00
Matei Zaharia e8df4bbd40 Added more SBT stuff to gitignore 2011-02-08 17:06:07 -08:00
Matei Zaharia 26b77aece9 Increased SBT mem to 700 MB so that unit tests run more nicely 2011-02-08 17:03:28 -08:00
Matei Zaharia 99f3f23efa Changed default shuffle to LocalFileShuffle because it's way faster for small files 2011-02-08 17:03:03 -08:00
Matei Zaharia f4f7aa2ab2 formatting 2011-02-08 16:39:17 -08:00
Matei Zaharia ee60aaa0f5 Added a pointer to wiki in readme 2011-02-08 16:38:10 -08:00
Mosharaf Chowdhury a12c0b6c00 Implemented rarest-first block request policy. Need to test it though. 2011-02-08 14:56:39 -08:00
Mosharaf Chowdhury 9b6d4074f9 Updating hasBlocks variable inside synchronized block 2011-02-05 12:14:41 -08:00
Matei Zaharia c1c766a93c Updated readme 2011-02-02 19:21:49 -08:00
Matei Zaharia 50df43bf7b Added SBT target for building a single JAR with Spark Core and its 
dependencies
 2011-02-02 19:08:14 -08:00
Matei Zaharia a11fe23017 Moved examples to spark.examples package 2011-02-02 16:30:27 -08:00
Matei Zaharia 82170608b1 Added IntelliJ's build directory to gitignore 2011-02-02 00:30:29 -08:00
Matei Zaharia ec28b607fd Merge branch 'master' into sbt 
Conflicts:
	Makefile
	core/src/main/java/spark/compress/lzf/LZF.java
	core/src/main/java/spark/compress/lzf/LZFInputStream.java
	core/src/main/java/spark/compress/lzf/LZFOutputStream.java
	core/src/main/native/spark_compress_lzf_LZF.c
	run
 2011-02-02 00:25:54 -08:00
Matei Zaharia 7f74ee99f6 Added support for IntelliJ IDEA 2011-02-02 00:08:13 -08:00
Matei Zaharia e5c4cd8a5e Made examples and core subprojects 2011-02-01 15:11:08 -08:00
Mosharaf Chowdhury fceae31877 Bug fix in selectSuitableSources: was changing the index value before updating a bitset using that same index variable. 2011-01-10 20:58:59 -08:00
Mosharaf Chowdhury ba74169162 Dummy log line added in SparkContext for #45 2011-01-06 21:21:35 -08:00
Mosharaf Chowdhury e76b4f7155 Turned OFF some log messages due to #45. 2011-01-06 17:07:14 -08:00
Mosharaf Chowdhury f5ba0b49c6 Bug fix + default config param values update. 
Updated some default config parameters for broadcast parameters.
Fixed a concurrency bug in the variable that keeps track of how many receivers have finished.
 2011-01-06 15:44:24 -08:00
Mosharaf Chowdhury 2fefbe17e4 TreeBroadcast is an extended version of ChainedBroadcast with customizable maxDegree per node. maxDegree = 1 is ChainedBroadcast. 2010-12-28 20:18:49 -08:00
Mosharaf Chowdhury 5540a99ab7 ChainedBroadcast is also reading masterHostAddress from config file until #42 is resolved. 2010-12-28 18:53:59 -08:00
Mosharaf Chowdhury 20eec59f04 Bug fix + formatting. 2010-12-25 22:54:46 -08:00
Mosharaf Chowdhury ba71b61e40 Reading masterHostAddress from config file until #42 has been resolved. 2010-12-25 10:22:22 -08:00
Mosharaf Chowdhury 5c5d767bc1 Modified MultiBroadcastTest. 2010-12-18 10:40:00 -08:00
Mosharaf Chowdhury d18d08ec9d Added a new BroadcastTest in the examples where 2 broadcasts are required. Should be used to experiment how multiple broadcasts work. 2010-12-17 10:43:49 -08:00
Matei Zaharia 817e722321 Merge branch 'master' of github.com:mesos/spark 2010-12-15 19:40:35 -08:00
Matei Zaharia 14c29c1b14 Fixed import 2010-12-15 19:40:27 -08:00
Mosharaf Chowdhury 5cafdd7ba2 Removed some unused imports from Broadcast.scala 2010-12-15 19:11:23 -08:00
Mosharaf Chowdhury be0ce57de2 - Fixed an compilation error due to wrong 'import' of legacy lzf libraries in DfsBroadcast.scala 
- Updated to use ning libraries.
 - Passes all unit tests
 2010-12-15 18:34:27 -08:00
Matei Zaharia 5c222dbe28 Merge branch 'master' into mos-bt 
Conflicts:
	src/scala/spark/Broadcast.scala
 2010-12-15 10:57:39 -08:00
Mosharaf Chowdhury 0a5c24ae3d - Default broadcast mechanism is set to DfsBroadcast 
- Configuration parameters are renamed to follow our convention
 - Master now automatically supplies its hostAddress instead of reading from config file
 - sendBroadcast has been removed from the Broadcast trait
 2010-12-13 14:36:39 -08:00
Timothy Hunter 34395730db Someone forgot to pass the parameters: fixes SPARK_MEM set from main script but not passed to executor. 2010-12-12 13:30:49 -08:00
Matei Zaharia 0d895ba636 Added BSD license 2010-12-07 10:32:17 -08:00
Mosharaf Chowdhury 06dc4a5148 - Removed config files from git's control. 
- Changed DfsShuffle to default in RDD.scala.
 2010-12-07 10:17:47 -08:00
Joshua Hartman 799c1b19f5 Adding license file for compress-lzf 2010-12-07 08:30:29 -08:00
Joshua Hartman 2fb849502f Replacing the native lzf compression code with the ning open-source compress-lzf library. (Apache 2.0 liscense) 2010-12-05 21:20:15 -08:00
Mosharaf Chowdhury 1013a01469 Made the root Broadcast object initilization method 'synchronized'. 2010-12-02 02:36:21 -08:00
Mosharaf Chowdhury a69638d98d Removed deprecated repl file ClassServer.scala 2010-11-30 18:45:04 -08:00
Mosharaf Chowdhury b7dda4c5bc Merge branch 'multi-tracker' into mos-bt 
Conflicts:
	conf/java-opts
	src/scala/spark/Broadcast.scala
	src/scala/spark/DfsBroadcast.scala
	src/scala/spark/SparkContext.scala
 2010-11-30 18:29:38 -08:00
Mosharaf Chowdhury 815ecd349a Made Broadcast Pluggable. Finally! 2010-11-30 18:08:49 -08:00
Mosharaf Chowdhury ea28a30e36 Makes transfer faster for small swarms (with size < MaxRxPeers) 2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury 191af65fb7 - Using the new Cache implementation. 2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury c9cad03c31 - Using the new Cache implementation. 
- Removed unused code related to dualMode (deprecated).
 2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury 73714da568 Made LocalFileShuffle to be the default. 2010-11-29 19:19:21 -08:00