ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
29993 commits 6 branches 181 tags 397 MiB
Commit graph

29993 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury f82cc17bc5 UseHttpPipelining option is brought back in. It works! 2010-12-07 10:07:30 -08:00
Joshua Hartman 799c1b19f5 Adding license file for compress-lzf 2010-12-07 08:30:29 -08:00
Joshua Hartman 2fb849502f Replacing the native lzf compression code with the ning open-source compress-lzf library. (Apache 2.0 liscense) 2010-12-05 21:20:15 -08:00
Mosharaf Chowdhury 7e2d72c328 Multiple connections created at a time. No upper limit on the server side though. 2010-12-04 18:55:55 -08:00
Mosharaf Chowdhury c6df327dd7 Updated logging format. 2010-12-04 16:41:13 -08:00
Mosharaf Chowdhury 7df20d681a Combined MaxRxPeers and MaxTxPeers to a single config parameter MaxConnections 2010-12-04 14:37:16 -08:00
Mosharaf Chowdhury b1745b3103 Removed an unnecessary byte array in the middle. Probabaly will have to bring it back if we do block level data movement. 2010-12-04 13:55:25 -08:00
Mosharaf Chowdhury 3a671ce989 Config parameters are in place. Good to go (I think) 2010-12-04 10:59:06 -08:00
Mosharaf Chowdhury 476a216d9d Parallel is working. Need to fix/finalize some config parameters. 2010-12-04 02:05:41 -08:00
Mosharaf Chowdhury c546c299bc combining is happening inside the thread. Its still synchronized though. 2010-12-04 00:59:25 -08:00
Mosharaf Chowdhury 0d7ca7751e Bug fixes. Not yet parallel. 2010-12-04 00:06:47 -08:00
Mosharaf Chowdhury 52086cef32 Building blocks are in place. Still not pulling parallely though. 2010-12-03 20:29:39 -08:00
Mosharaf Chowdhury 540a41163f UseHttpPipelining is 'true' by default. 2010-12-02 19:56:17 -08:00
Mosharaf Chowdhury 1013a01469 Made the root Broadcast object initilization method 'synchronized'. 2010-12-02 02:36:21 -08:00
Mosharaf Chowdhury 0de859fbe2 Enabling/disabling HTTP pipelining is a config option now. Performance tradeoffs are not obvious yet. 2010-12-02 02:32:44 -08:00
Mosharaf Chowdhury a69638d98d Removed deprecated repl file ClassServer.scala 2010-11-30 18:45:04 -08:00
Mosharaf Chowdhury b7dda4c5bc Merge branch 'multi-tracker' into mos-bt 
Conflicts:
	conf/java-opts
	src/scala/spark/Broadcast.scala
	src/scala/spark/DfsBroadcast.scala
	src/scala/spark/SparkContext.scala
 2010-11-30 18:29:38 -08:00
Mosharaf Chowdhury 815ecd349a Made Broadcast Pluggable. Finally! 2010-11-30 18:08:49 -08:00
Mosharaf Chowdhury ea28a30e36 Makes transfer faster for small swarms (with size < MaxRxPeers) 2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury 191af65fb7 - Using the new Cache implementation. 2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury c9cad03c31 - Using the new Cache implementation. 
- Removed unused code related to dualMode (deprecated).
 2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury 73714da568 Made LocalFileShuffle to be the default. 2010-11-29 19:19:21 -08:00
Mosharaf Chowdhury c62f2fefa5 Made LocalFileShuffle to be the default. 2010-11-29 19:17:01 -08:00
Mosharaf Chowdhury 15f1f6865d Variable renaming. 2010-11-29 14:15:36 -08:00
Mosharaf Chowdhury bb3e7fbf61 Variable renaming. 2010-11-29 14:13:35 -08:00
Mosharaf Chowdhury 1157380f21 Removed unused third party libraries. 2010-11-29 12:21:46 -08:00
Mosharaf Chowdhury 1f7fcb48be Merge branch 'master' into mos-bt 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/Executor.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/MesosScheduler.scala
	src/scala/spark/RDD.scala
	src/scala/spark/SparkContext.scala
	src/scala/spark/Split.scala
	src/scala/spark/Utils.scala
	src/scala/spark/repl/SparkInterpreter.scala
	third_party/mesos.jar
 2010-11-29 12:10:06 -08:00
Mosharaf Chowdhury 9e93f8df12 Moved broadcast implementations to separate files. Ready to merge with 'master' 2010-11-29 11:43:20 -08:00
Mosharaf Chowdhury 8d775448c2 - Resolved some of the simpler TODOs related to different timeout and wait periods. 
- Removed unused code.
 2010-11-29 11:20:04 -08:00
Mosharaf Chowdhury 8494b3a4f9 - Added log messages for benchmarking. 
- Added GroupByTest.scala for benchmarking.
 2010-11-27 23:51:43 -08:00
Mosharaf Chowdhury 19a6b194a9 Merge branch 'master' into multi-tracker 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/Executor.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/MesosScheduler.scala
	src/scala/spark/RDD.scala
	src/scala/spark/SparkContext.scala
	src/scala/spark/Split.scala
	src/scala/spark/Utils.scala
	src/scala/spark/repl/SparkInterpreter.scala
	third_party/mesos.jar
 2010-11-27 01:27:20 -08:00
Mosharaf Chowdhury e4b8db45ae - Moved DaemonThreadPool factory methods to the Broadcast Object. 
- Reflection-based broadcast class loading is still not working.
 2010-11-27 00:53:57 -08:00
Mosharaf Chowdhury c25914228c Moved ChaninedStreaming to a separate file and renamed to ChainedBroadcast. 2010-11-26 23:15:33 -08:00
Mosharaf Chowdhury 98542f81bb - Removed dual Mode 
- Removed ununsed speed-related stuff
- Moved (H)DfsBroadcast to its own file
 2010-11-26 22:58:35 -08:00
Mosharaf Chowdhury 9d0111659b Solved compilation issue. Should've been more careful to do a clean compilation before committing. 2010-11-26 22:10:38 -08:00
Mosharaf Chowdhury 9c8a65b920 Removed everything related to SplitStreaming... 2010-11-26 22:06:43 -08:00
Mosharaf Chowdhury 830496b901 Updated some comments. 2010-11-25 23:49:57 -08:00
Mosharaf Chowdhury 690b55917d Slaves weren't printing hostAddress. Fixed. 2010-11-18 10:53:34 -08:00
Mosharaf Chowdhury 2de1664214 Slaves weren't logging hostAddress. Fixed. 2010-11-18 10:51:57 -08:00
Mosharaf Chowdhury dfa6540fdd Everyone logs local hostAddress. 2010-11-18 09:58:46 -08:00
Mosharaf Chowdhury 1f2662d45a Everyone logs their local hostAddress. 2010-11-18 09:55:19 -08:00
Matei Zaharia dcfa2ce83b Further improvements -- build native stuff in target directory and add a 
test-report target for XML test reports
 2010-11-14 00:46:19 -08:00
Matei Zaharia e86b620f9e Fixed some more stuff (Eclispe target and native build) 2010-11-13 22:46:00 -08:00
Matei Zaharia 89fcd96702 Initial work to get Spark compiling with SBT 0.7.5 RC0 2010-11-13 22:07:08 -08:00
Mosharaf Chowdhury c3f354c6fc Guide now select random peers to select sources instead of rolling statically. 
Random is good. Deterministic is bad. Well, mostly.
 2010-11-13 20:18:48 -08:00
Matei Zaharia b7574201d5 Added some stuff to .gitignore 2010-11-13 18:41:35 -08:00
Matei Zaharia f8ea98d989 Remove -unchecked compiler parameter 2010-11-13 18:39:07 -08:00
Matei Zaharia af74217c96 Undo JLine fix that turns out to only be needed when buildr is running 
on JRuby. This is quite ugly: JRuby has its own version of JLine which
is older than Scala's, and JLine changed API in such a way that code
written for the new version won't compile with the old one and vice
versa. Sadly, this might be a reason to drop buildr, unless we can
package a JRuby with it that uses the right version, or we ask people to
only use the C Ruby version of buildr (which doesn't work on OS X right
now!)
 2010-11-13 18:38:50 -08:00
Matei Zaharia b84769a107 Modified project structure to work with buildr 2010-11-13 17:18:05 -08:00
Matei Zaharia f8966ffc11 Added a shuffle test with negative hash codes for some keys (this was a bug earlier) 2010-11-12 16:18:45 -08:00