ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
17081 commits 6 branches 181 tags 397 MiB
Commit graph

17081 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia 8c95a85438 Use Runtime.maxMemory instead of Runtime.totalMemory in 
BoundedMemoryCache, in case the JVM was not started with its initial
heap size equaling its maximum one (-Xms == -Xmx).
 2012-03-30 13:39:35 -04:00
Matei Zaharia 03d5b3b48d Use Runtime.maxMemory instead of Runtime.totalMemory in 
BoundedMemoryCache, in case the JVM was not started with its initial
heap size equaling its maximum one (-Xms == -Xmx).
 2012-03-30 13:38:19 -04:00
Matei Zaharia 95fb1a16b8 Use Mesos 0.9 RC3 JAR and protobuf 2.4.1 2012-03-30 11:38:49 -04:00
Matei Zaharia dfa3b6b544 Fixes to work with the very latest Mesos 0.9 API 2012-03-29 22:12:35 -04:00
Matei Zaharia 4d52cc6738 Merge branch 'master' into mesos-0.9 2012-03-29 21:29:39 -04:00
Reynold Xin d46f662c5b Merge branch 'master' of github.com:mesos/spark 2012-03-29 15:22:17 -07:00
Reynold Xin 42dcdbcb2f Removed the extra spaces in OrderedRDDFunctions and SortedRDD. 2012-03-29 15:21:57 -07:00
Matei Zaharia ca5c19c1ba Remove dependency on Akka 2012-03-29 01:03:34 -04:00
Reynold Xin 90418b70ff Added sbt-assembly for spark-repl project so we can generate an 
assembled jar for Shark.
 2012-03-22 18:46:31 -07:00
Matei Zaharia ca64a7ae03 Documentation 2012-03-17 13:51:29 -07:00
Matei Zaharia 36c7db72bc Documentation 2012-03-17 13:49:55 -07:00
Matei Zaharia 08cda89e8a Further fixes to how Mesos is found and used 2012-03-17 13:39:14 -07:00
Matei Zaharia 3c3fdf6eca Merge branch 'master' into mesos-0.9 2012-03-17 13:09:21 -07:00
Matei Zaharia c7af538ac1 Some fixes to sorting for when the RDD has fewer elements than the 
number of partitions we ask to partition it into. Also, removed a test
that was taking way too long to run.
 2012-03-17 13:08:36 -07:00
Matei Zaharia a099a63a8a Initial work to make Spark compile with Mesos 0.9 and Hadoop 1.0 2012-03-17 12:31:34 -07:00
Matei Zaharia a5e2b6a6bd Merge pull request #112 from cengle/master 
Changed HadoopRDD to get key and value containers from the RecordReader instead of through reflection
 2012-03-06 13:38:32 -08:00
Matei Zaharia 97eee50825 Fixes a nasty bug that could happen when tasks fail, because calling 
wait() with a timeout of 0 on a Java object means "wait forever".
 2012-03-01 13:43:17 -08:00
Cliff Engle dd68cb6099 Get key and value container from RecordReader 2012-02-29 16:33:23 -08:00
Matei Zaharia 1e10df0a46 Merge pull request #111 from alupher/master 
Adding sorting to RDDs
 2012-02-24 15:50:14 -08:00
Antonio 0d93d95bcf Removed unnecessary import 2012-02-21 19:57:12 -08:00
Antonio 2990298f71 Added sorting testing suite 2012-02-21 19:54:21 -08:00
Matei Zaharia aa04f87cd2 Added support for parallel execution of jobs in DAGScheduler. 2012-02-19 22:50:23 -08:00
Antonio 620798161b Added fixes to sorting 2012-02-13 00:07:39 -08:00
Matei Zaharia 2587ce1690 Fixed a deadlock that occured with MesosScheduler due to an earlier 
synchronization change
 2012-02-11 21:22:45 -08:00
Antonio e93f622665 Added sorting by key for pair RDDs 2012-02-11 00:56:28 -08:00
Matei Zaharia 98f008b721 Formatting fixes 2012-02-10 10:52:03 -08:00
Matei Zaharia 7660a8b12f Merge branch 'formatting' 
Conflicts:
	core/src/main/scala/spark/DAGScheduler.scala
	core/src/main/scala/spark/SimpleShuffleFetcher.scala
	core/src/main/scala/spark/SparkContext.scala
 2012-02-10 10:42:14 -08:00
haoyuan 194c42ab79 Code format. 2012-02-10 08:19:53 -08:00
Matei Zaharia 8f5ed51234 Delete Spark's temporary directories when the JVM exits. 2012-02-09 22:58:24 -08:00
Matei Zaharia c0a0df3285 Made the default cache BoundedMemoryCache, and reduced its default size 2012-02-09 22:32:02 -08:00
Matei Zaharia a766780f4c Added some tests for multithreaded access to Spark. 2012-02-09 22:27:53 -08:00
Matei Zaharia 0e93891d3d Replaced LocalFileShuffle with a non-singleton ShuffleManager class 
and made DAGScheduler automatically set SparkEnv.
 2012-02-09 22:14:56 -08:00
haoyuan 445e0bb1b5 Format the code a bit mroe. 2012-02-09 15:50:26 -08:00
haoyuan 651932e703 Format the code as coding style agreed by Matei/TD/Haoyuan 2012-02-09 13:26:23 -08:00
Matei Zaharia e02dc83a5b IO optimizations 2012-02-06 20:40:39 -08:00
Matei Zaharia c40e766368 Use java.util.HashMap in shuffles 2012-02-06 19:20:25 -08:00
Matei Zaharia d6ec664b48 Add dependency on fastutil and update Guava 2012-02-06 15:37:27 -08:00
Matei Zaharia b267175ab5 Synchronization fix in case SparkContext is used from multiple threads. 2012-02-06 14:28:18 -08:00
haoyuan b72d93a0da Test commit 2012-02-06 09:58:06 -08:00
Matei Zaharia 43a3335090 Simplifying test 2012-02-05 22:46:51 -08:00
Matei Zaharia 7449ecfb7e Merge branch 'master' of github.com:mesos/spark 2012-01-31 00:33:24 -08:00
Matei Zaharia 100e800782 Some fixes to the examples (mostly to use functional API) 2012-01-31 00:33:18 -08:00
Matei Zaharia 72d2489b6d Merge pull request #108 from patelh/master 
Added immutable map registration in kryo serializer
 2012-01-30 16:31:12 -08:00
Hiral Patel b47952342e Add register immutable map to kryo serializer 2012-01-26 15:24:20 -08:00
Matei Zaharia fabcc82528 Merge pull request #103 from edisontung/master 
Made improvements to takeSample. Also changed SparkLocalKMeans to SparkKMeans
 2012-01-13 19:20:03 -08:00
Matei Zaharia fd5581a0d3 Fixed a failure recovery bug and added some tests for fault recovery. 2012-01-13 19:17:27 -08:00
Matei Zaharia eb05154b7a Fixed a failure recovery bug and added some tests for fault recovery. 2012-01-13 19:08:25 -08:00
Edison Tung 1ecc221f84 Fixed bugs 
I've fixed the bugs detailed in the diff. One of the bugs was already
fixed on the local file (forgot to commit).
 2012-01-09 11:59:52 -08:00
Matei Zaharia e269f6f7ea Register RDDs with the MapOutputTracker even if they have no partitions. 
Fixes #105.
 2012-01-05 15:59:20 -05:00
Matei Zaharia 5fd101d79e Add dependency on Akka and Netty 2011-12-15 13:21:14 +01:00