ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
6525 commits 6 branches 181 tags 397 MiB
Commit graph

6525 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury ea28a30e36 Makes transfer faster for small swarms (with size < MaxRxPeers) 2010-11-30 16:13:56 -08:00
Mosharaf Chowdhury 191af65fb7 - Using the new Cache implementation. 2010-11-29 20:35:47 -08:00
Mosharaf Chowdhury c9cad03c31 - Using the new Cache implementation. 
- Removed unused code related to dualMode (deprecated).
 2010-11-29 20:30:38 -08:00
Mosharaf Chowdhury 73714da568 Made LocalFileShuffle to be the default. 2010-11-29 19:19:21 -08:00
Mosharaf Chowdhury c62f2fefa5 Made LocalFileShuffle to be the default. 2010-11-29 19:17:01 -08:00
Mosharaf Chowdhury 15f1f6865d Variable renaming. 2010-11-29 14:15:36 -08:00
Mosharaf Chowdhury bb3e7fbf61 Variable renaming. 2010-11-29 14:13:35 -08:00
Mosharaf Chowdhury 1157380f21 Removed unused third party libraries. 2010-11-29 12:21:46 -08:00
Mosharaf Chowdhury 1f7fcb48be Merge branch 'master' into mos-bt 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/Executor.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/MesosScheduler.scala
	src/scala/spark/RDD.scala
	src/scala/spark/SparkContext.scala
	src/scala/spark/Split.scala
	src/scala/spark/Utils.scala
	src/scala/spark/repl/SparkInterpreter.scala
	third_party/mesos.jar
 2010-11-29 12:10:06 -08:00
Mosharaf Chowdhury 9e93f8df12 Moved broadcast implementations to separate files. Ready to merge with 'master' 2010-11-29 11:43:20 -08:00
Mosharaf Chowdhury 8d775448c2 - Resolved some of the simpler TODOs related to different timeout and wait periods. 
- Removed unused code.
 2010-11-29 11:20:04 -08:00
Mosharaf Chowdhury 8494b3a4f9 - Added log messages for benchmarking. 
- Added GroupByTest.scala for benchmarking.
 2010-11-27 23:51:43 -08:00
Mosharaf Chowdhury 19a6b194a9 Merge branch 'master' into multi-tracker 
Conflicts:
	Makefile
	run
	src/scala/spark/Broadcast.scala
	src/scala/spark/Executor.scala
	src/scala/spark/HdfsFile.scala
	src/scala/spark/MesosScheduler.scala
	src/scala/spark/RDD.scala
	src/scala/spark/SparkContext.scala
	src/scala/spark/Split.scala
	src/scala/spark/Utils.scala
	src/scala/spark/repl/SparkInterpreter.scala
	third_party/mesos.jar
 2010-11-27 01:27:20 -08:00
Mosharaf Chowdhury e4b8db45ae - Moved DaemonThreadPool factory methods to the Broadcast Object. 
- Reflection-based broadcast class loading is still not working.
 2010-11-27 00:53:57 -08:00
Mosharaf Chowdhury c25914228c Moved ChaninedStreaming to a separate file and renamed to ChainedBroadcast. 2010-11-26 23:15:33 -08:00
Mosharaf Chowdhury 98542f81bb - Removed dual Mode 
- Removed ununsed speed-related stuff
- Moved (H)DfsBroadcast to its own file
 2010-11-26 22:58:35 -08:00
Mosharaf Chowdhury 9d0111659b Solved compilation issue. Should've been more careful to do a clean compilation before committing. 2010-11-26 22:10:38 -08:00
Mosharaf Chowdhury 9c8a65b920 Removed everything related to SplitStreaming... 2010-11-26 22:06:43 -08:00
Mosharaf Chowdhury 830496b901 Updated some comments. 2010-11-25 23:49:57 -08:00
Mosharaf Chowdhury 690b55917d Slaves weren't printing hostAddress. Fixed. 2010-11-18 10:53:34 -08:00
Mosharaf Chowdhury 2de1664214 Slaves weren't logging hostAddress. Fixed. 2010-11-18 10:51:57 -08:00
Mosharaf Chowdhury dfa6540fdd Everyone logs local hostAddress. 2010-11-18 09:58:46 -08:00
Mosharaf Chowdhury 1f2662d45a Everyone logs their local hostAddress. 2010-11-18 09:55:19 -08:00
Matei Zaharia dcfa2ce83b Further improvements -- build native stuff in target directory and add a 
test-report target for XML test reports
 2010-11-14 00:46:19 -08:00
Matei Zaharia e86b620f9e Fixed some more stuff (Eclispe target and native build) 2010-11-13 22:46:00 -08:00
Matei Zaharia 89fcd96702 Initial work to get Spark compiling with SBT 0.7.5 RC0 2010-11-13 22:07:08 -08:00
Mosharaf Chowdhury c3f354c6fc Guide now select random peers to select sources instead of rolling statically. 
Random is good. Deterministic is bad. Well, mostly.
 2010-11-13 20:18:48 -08:00
Matei Zaharia b7574201d5 Added some stuff to .gitignore 2010-11-13 18:41:35 -08:00
Matei Zaharia f8ea98d989 Remove -unchecked compiler parameter 2010-11-13 18:39:07 -08:00
Matei Zaharia af74217c96 Undo JLine fix that turns out to only be needed when buildr is running 
on JRuby. This is quite ugly: JRuby has its own version of JLine which
is older than Scala's, and JLine changed API in such a way that code
written for the new version won't compile with the old one and vice
versa. Sadly, this might be a reason to drop buildr, unless we can
package a JRuby with it that uses the right version, or we ask people to
only use the C Ruby version of buildr (which doesn't work on OS X right
now!)
 2010-11-13 18:38:50 -08:00
Matei Zaharia b84769a107 Modified project structure to work with buildr 2010-11-13 17:18:05 -08:00
Matei Zaharia f8966ffc11 Added a shuffle test with negative hash codes for some keys (this was a bug earlier) 2010-11-12 16:18:45 -08:00
Matei Zaharia d0a9966555 Unit tests for shuffle operations. Fixes #33. 2010-11-12 16:12:14 -08:00
Mosharaf Chowdhury 9915fd22c8 Added BroadcastBT.EndGameFraction config option to control when to start end game. 2010-11-12 15:27:47 -08:00
Mosharaf Chowdhury c6874d4370 Talk to peers even if current knowledge says they dont have anything to send. 2010-11-12 14:36:34 -08:00
Mosharaf Chowdhury 0adb12e5aa Merge branch 'mos-bt' into mos-bt-ask-for-block 2010-11-11 14:07:02 -08:00
Mosharaf Chowdhury 23ebac8c30 This reverts the addToListOfSources methods of commit 642098c211. 2010-11-11 14:05:38 -08:00
Mosharaf Chowdhury 88d08b7a89 Bug fix. 2010-11-10 17:04:14 -08:00
Mosharaf Chowdhury eb70f6a36e Receivers explicitly ask for blocks to send. 
The Random generator must be updated so that everyone does not ask for the same block during the initial phase.
 2010-11-10 16:47:45 -08:00
Mosharaf Chowdhury d7579e743a Style update: all functions that return void should have a Unit return type. 2010-11-10 10:32:29 -08:00
Mosharaf Chowdhury 642098c211 Bug fixes and/or minor optimization. 2010-11-09 23:59:06 -08:00
Mosharaf Chowdhury b58bbadbfb All daemon => Fast shutdown :) 2010-11-09 17:05:19 -08:00
Mosharaf Chowdhury 76610e842b Removed some unnecessary stuff. 2010-11-09 16:59:32 -08:00
Mosharaf Chowdhury 50612ab3a9 Every thread is a daemon thread => Program shuts down ASAP once its done. 
Some log messages added to blank exception handlers.
 2010-11-09 16:44:42 -08:00
Matei Zaharia 7b25ab87af Added options for using an external HTTP server with LocalFileShuffle 2010-11-09 13:46:30 -08:00
Mosharaf Chowdhury 1820634dbf Bug fix: java-opts had wrong (non-capitalized) variable names. 2010-11-08 18:32:17 -08:00
Matei Zaharia 504f839c65 Removed unnecessary collectAsMap 2010-11-08 08:49:42 -08:00
Matei Zaharia 9d3f05a990 Made shuffle algorithm pluggable and added LocalFileShuffle. 2010-11-08 00:46:12 -08:00
Mosharaf Chowdhury 6cb76d9708 Commented out some cluttering log messages. 2010-11-07 19:16:03 -08:00
Mosharaf Chowdhury cbce44304c Added exception handling to 'sendStopBroadcastNotifications' 2010-11-06 19:37:27 -07:00