ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
345 commits 6 branches 181 tags 397 MiB
Commit graph

345 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 23b1c309fb Added pipe() operation on RDDs for mapping through a shell command. 2011-06-19 23:05:19 -07:00
Matei Zaharia a413b8e59d Merge pull request #59 from ijuma/master 
Move managedStyle to SparkProject
 2011-06-07 00:41:50 -07:00
Ismael Juma 1ad4dcd3de Move managedStyle to SparkProject. 
I had added it to DepJar by mistake.
 2011-06-02 14:06:54 +01:00
Matei Zaharia 9bb448a151 Catch Throwable instead of Exception in LocalScheduler and Executor. Fixes #57. 2011-06-01 11:45:47 -07:00
Matei Zaharia 850fe3274e Make the runJob API public. Fixes #56. 2011-06-01 11:38:44 -07:00
Matei Zaharia 5166d76843 Ensure logging is initialized before spawning any threads to fix issue #45 2011-05-31 23:47:32 -07:00
Matei Zaharia 96daa31a01 Pass quoted arguments properly to run 2011-05-31 23:34:16 -07:00
Matei Zaharia 7862995569 Give SBT a bit more memory so it can do a update / compile / test in one JVM 2011-05-31 23:33:47 -07:00
Matei Zaharia 8b0390d344 Instantiate NullWritable properly in HadoopFile 2011-05-30 23:54:14 -07:00
Matei Zaharia c501cff924 Executor was looking for the wrong constructor for ExecutorClassLoader 2011-05-29 16:15:59 -07:00
Matei Zaharia 50ac1d2a40 Merge remote-tracking branch 'ijuma/issue51' 2011-05-29 15:41:12 -07:00
Ismael Juma 0c62ee4321 Depend on jetty-server in compile scope and upgrade to 7.4.2. 
As Matei described: "We're using Jetty to run an HTTP server, not to embed Spark
in a webapp"
 2011-05-29 20:12:50 +01:00
Matei Zaharia 22c8b84d8b Merge pull request #53 from ijuma/master 
Use explicit asInstanceOf instead of misleading unchecked pattern matching.
 2011-05-27 10:20:54 -07:00
Ismael Juma e3b323321d Use ManagedStyle.Maven. 2011-05-27 14:56:01 +01:00
Ismael Juma 3a6b0b8a57 Publish javadoc and sources. 2011-05-27 14:55:51 +01:00
Ismael Juma 59f1f42a9a Update run to work with SBT managed dependencies and the newly introduced repl module. 2011-05-27 11:22:59 +01:00
Ismael Juma 3af6003c87 Update sbt to 0.7.7. 2011-05-27 11:22:59 +01:00
Ismael Juma 1396678baa Move REPL classes to separate module. 2011-05-27 11:22:50 +01:00
Ismael Juma 3e8114ddbd Change project.organization to org.spark-project to fit Maven convention. 2011-05-27 11:22:10 +01:00
Ismael Juma 7b7dfdb085 Set project.version to 0.3-SNAPSHOT. 2011-05-27 11:22:10 +01:00
Ismael Juma 051da8b4ad Delete liblzf from lib as it's no longer used. 2011-05-27 11:22:10 +01:00
Ismael Juma ae1a1f91f1 Remove several dependencies from git and configure them as SBT managed dependencies. 
Upgrade some of the dependencies while at it.
 2011-05-27 11:22:01 +01:00
Ismael Juma 164ef4c751 Use explicit asInstanceOf instead of misleading unchecked pattern matching. 
Also enable -unchecked warnings in SBT build file.
 2011-05-27 07:57:10 +01:00
Matei Zaharia cfbe2da1a6 Merge pull request #52 from ijuma/master 
Fix deprecations when compiled with Scala 2.8.1
 2011-05-26 23:53:10 -07:00
Ismael Juma 89c8ea2bb2 Replace deprecated - and -- with suggested filterNot (which is uglier). 2011-05-26 22:22:37 +01:00
Ismael Juma 94f05683bd Replace deprecated first with head. 2011-05-26 22:13:41 +01:00
Ismael Juma 26000af4fa Replace deprecated fromFunction with either tabulate or fill. 
tabulate used if indexed used by function and fill otherwise.
 2011-05-26 22:12:11 +01:00
Ismael Juma 0b6a862b68 Use math instead of Math as the latter is deprecated. 2011-05-26 22:06:36 +01:00
Ismael Juma 1f27d94c48 Use Array.iterator instead of Iterator.fromArray as the latter is deprecated. 2011-05-26 22:04:42 +01:00
Ismael Juma 1993a8e556 Use += instead of + for mutable sequences as the latter is deprecated. 2011-05-26 21:59:48 +01:00
Ismael Juma 222729171e Upgrade sbt-idea to 0.4.0. 2011-05-26 21:48:08 +01:00
Matei Zaharia f707856522 Removed java-opts.template 2011-05-24 15:59:01 -07:00
Matei Zaharia cec427e777 Fixed a bug with preferred locations having changed meaning in new RDDs 2011-05-22 17:12:29 -07:00
Matei Zaharia 4c888b2933 Fix queue type for executor 2011-05-22 16:42:05 -07:00
Matei Zaharia bea3a33012 doc tweak 2011-05-22 16:03:41 -07:00
Matei Zaharia 9bde5a54cb class loader fix 2011-05-22 16:00:41 -07:00
Matei Zaharia 91c07a33d9 Various fixes to serialization 2011-05-21 22:50:08 -07:00
Matei Zaharia f61b61c4ac Merge branch 'master' into new-rdds 2011-05-21 21:25:58 -07:00
Matei Zaharia 24a1e7f838 Scheduler can now recover from lost map outputs 2011-05-20 00:19:53 -07:00
Matei Zaharia 82329b0b28 Updated scheduler to support running on just some partitions of final RDD 2011-05-19 12:47:09 -07:00
Matei Zaharia 328e51b693 Various minor fixes 2011-05-19 11:19:25 -07:00
Matei Zaharia fd1d255821 Stop objectifying various trackers, caches, etc. 2011-05-17 12:41:13 -07:00
Matei Zaharia 4db50e26c7 Fixed unit tests by making them clean up the SparkContext after use and 
thus clean up the various singletons (RDDCache, MapOutputTracker, etc).
This isn't perfect yet (ideally we shouldn't use singleton objects at
all) but we can fix that later.
 2011-05-13 12:03:58 -07:00
Matei Zaharia aca8150c52 Ensure that AddedToCache messages make it home before tasks finish 2011-05-13 11:43:52 -07:00
Matei Zaharia 16c886a581 Optimization for count() 2011-05-13 10:41:34 -07:00
Matei Zaharia 4b1f0f1ce4 Merge pull request #48 from ankurdave/bagel-new 
Bagel: Large-scale graph processing on Spark
 2011-05-12 21:34:38 -07:00
Ankur Dave f40a0898a7 Rename bagel to spark.bagel and Pregel to Bagel 2011-05-09 15:23:21 -07:00
Matei Zaharia 7e20648914 Upgraded to SBT 0.7.5 2011-05-09 14:48:39 -07:00
Matei Zaharia 4bedf5b13a Merge pull request #47 from ankurdave/cache-to-disk 
Merging in Ankur's code for a cache that spills to disk
 2011-05-09 14:22:56 -07:00
Ankur Dave c1104058c6 Move shortest path and PageRank to bagel.examples 2011-05-03 18:53:58 -07:00