ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
948 commits 6 branches 181 tags 397 MiB
Commit graph

948 commits

This branch
This branch
All branches
Author SHA1 Message Date
Andy Konwinski 49e98500a9 Updated base README to point to documentation site instead of wiki, updated 
docs/README.md to describe use of Jekyll, and renmaed things to make them
more consistent with the lower-case-with-hyphens convention.
 2012-09-12 13:03:43 -07:00
Andy Konwinski 16da942d66 Adding docs directory containing documentation currently on the wiki 
which can be compiled via jekyll, using the command `jekyll`. To compile
and run a local webserver to serve the doc as a website, run
`jekyll --server`.
 2012-09-12 13:03:43 -07:00
Matei Zaharia a29ac5f9cf Merge pull request #195 from dennybritz/feature/fileserver 
Spark HTTP FileServer
 2012-09-11 17:04:17 -07:00
Denny 5e4076e3f2 Merge branch 'dev' into feature/fileserver 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
 2012-09-11 16:57:17 -07:00
Denny 77873d2c8e Formatting 2012-09-11 16:51:46 -07:00
Denny 24b9b37314 Subclass URLClassLoader instead of using reflection 2012-09-11 16:51:08 -07:00
Denny 31c53e917d Use stageId as index for fileSet caches. 2012-09-11 16:10:45 -07:00
Matei Zaharia 943df48348 Merge branch 'dev' of github.com:mesos/spark into dev 2012-09-11 16:00:37 -07:00
Matei Zaharia 6d7f907e73 Manually merge pull request #175 by Imran Rashid 2012-09-11 16:00:06 -07:00
Matei Zaharia 4767ba5b7d Merge pull request #196 from rxin/dev 
Log entire exception (including stack trace) in BlockManagerWorker.
 2012-09-11 14:40:44 -07:00
Reynold Xin 7af7c79ce5 Updated the logError call from the previous commit to conform to 
logError API.
 2012-09-11 14:32:24 -07:00
Reynold Xin 38b9119c96 Log entire exception (including stack trace) in BlockManagerWorker. 2012-09-11 11:31:35 -07:00
Denny 4d3471dd07 Fix serialization bugs and added local cluster tests 2012-09-10 15:39:58 -07:00
Denny 94a7e82ba1 Merge branch 'dev' of git://github.com/mesos/spark into feature/fileserver 2012-09-10 12:54:20 -07:00
Denny b864c36a30 Dynamically adding jar files and caching fileSets. 2012-09-10 12:49:09 -07:00
Denny f275fb07da General FileServer 
A general fileserver for both JARs and regular files.
 2012-09-10 12:48:59 -07:00
Matei Zaharia a13780670d Added a unit test for local-cluster mode and simplified some of the code involved in that 2012-09-10 12:48:58 -07:00
Denny f2ac55840c Add shutdown hook to Executor Runner and execute code to shutdown local cluster in Scheduler Backend 2012-09-10 12:48:58 -07:00
Denny 9ead8ab14e Set SPARK_LAUNCH_WITH_SCALA=0 in Executor Runner 2012-09-10 12:48:58 -07:00
Denny 8bb3c73977 Renamed spark-cluster to spark-local. 2012-09-10 12:48:58 -07:00
Denny a367c20f49 Fix wrong counting 2012-09-10 12:48:57 -07:00
Denny 93fe331e6d Delete old DeployUtils. 2012-09-10 12:48:57 -07:00
Denny cf074f9c96 Renamed class. 2012-09-10 12:48:57 -07:00
Denny 3749f94184 Start a standalone cluster locally. 2012-09-10 12:48:57 -07:00
Matei Zaharia 995982b3c9 Added a unit test for local-cluster mode and simplified some of the code involved in that 2012-09-07 17:08:36 -07:00
Matei Zaharia 8d2fcc2832 Merge pull request #189 from dennybritz/feature/localcluster 
Simulating a Spark standalone cluster locally
 2012-09-07 15:43:43 -07:00
Denny 7ff9311add Add shutdown hook to Executor Runner and execute code to shutdown local cluster in Scheduler Backend 2012-09-07 14:09:12 -07:00
Denny 4e7b264cf7 Set SPARK_LAUNCH_WITH_SCALA=0 in Executor Runner 2012-09-07 11:39:44 -07:00
root c2da64409a Randomize the order of block fetches in getMultiple 2012-09-06 23:16:26 +00:00
Denny 886183e591 Renamed spark-cluster to spark-local. 2012-09-05 17:10:54 -07:00
Matei Zaharia 53a5681c8a Merge pull request #190 from rxin/dev 
Log cache add/remove messages in block manager.
 2012-09-05 16:41:52 -07:00
Reynold Xin c308fbcb79 Removed cache add/remove log messages from CacheTracker. 
Added log messages on BlockManagerMaster to reflect block add/remove.
Also did some minor cleanup of storage package code.
 2012-09-05 15:59:48 -07:00
Denny babbca0a2f Fix wrong counting 2012-09-04 22:04:18 -07:00
Denny 9326509f66 Delete old DeployUtils. 2012-09-04 21:15:23 -07:00
Denny 1588d4dbe6 Renamed class. 2012-09-04 21:13:25 -07:00
Denny 22dde6e020 Start a standalone cluster locally. 2012-09-04 20:56:30 -07:00
Matei Zaharia a842c63044 Minor formatting fixes 2012-09-03 16:24:00 -07:00
Matei Zaharia 2d6a629f8c Merge pull request #182 from HarveyFeng/dev-fetch 
Add a limit on the number of parallel fetches in the reduce stage
 2012-09-03 16:14:57 -07:00
Harvey 3076b038f4 Start fetching a remote block when a received remote block has been passed 
to the reduce function
 2012-09-01 12:01:35 -07:00
Matei Zaharia 389fb4cc54 End runJob() with a SparkException when a task fails too many times in 
one of the cluster schedulers.
 2012-08-31 17:47:43 -07:00
Matei Zaharia a480dec6b2 Deserialize multi-get results in the caller's thread. This fixes an 
issue with shared buffers in the KryoSerializer.
 2012-08-30 20:01:06 -07:00
Matei Zaharia bf3212615a Merge pull request #184 from rxin/dev 
Disable running combiners on map tasks when mergeCombiners function is not specified by the user.
 2012-08-30 14:12:40 -07:00
Reynold Xin a8a2a08a1a Added a test for testing map-side combine on/off switch. 2012-08-30 12:34:28 -07:00
Reynold Xin 5945bcdcc5 Added a new flag in Aggregator to indicate applying map side combiners. 2012-08-29 23:32:08 -07:00
Reynold Xin c68e820b2a Merge branch 'dev' of github.com:mesos/spark into dev 2012-08-29 23:01:19 -07:00
Reynold Xin 940869dfda Disable running combiners on map tasks when mergeCombiners function is 
not specified by the user.
 2012-08-29 23:00:02 -07:00
Matei Zaharia 47507d69d9 Made region used by spark-ec2 configurable. 2012-08-28 22:40:00 -07:00
Matei Zaharia bf2e9cb08e Fault tolerance and block store fixes discovered through streaming tests. 2012-08-27 23:07:50 -07:00
Matei Zaharia 291abc2c28 Merge pull request #181 from rxin/dev 
Removed the deserialization cache for ShuffleMapTask
 2012-08-27 22:38:22 -07:00
Reynold Xin 3a6a95dc24 Removed the deserialization cache for ShuffleMapTask because it was 
causing concurrency problems (some variables in Shark get set to null).
The cost of task deserialization on slaves is trivial compared with the
execution time of the task anyway.
 2012-08-27 22:33:15 -07:00