ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
13261 commits 6 branches 181 tags 397 MiB
Commit graph

13261 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia e2e71a1fb5 Merge remote-tracking branch 'public/dev' into dev 2012-07-28 20:26:59 -07:00
Matei Zaharia d7f089323a Fixed AccumulatorSuite to clean up SparkContext with BeforeAndAfter 2012-07-28 20:25:42 -07:00
Imran Rashid f7149c5e46 tasks cannot access value of accumulator 2012-07-28 20:16:17 -07:00
Imran Rashid 244cbbe33a one more minor cleanup to scaladoc 2012-07-28 20:16:10 -07:00
Imran Rashid 3b392c67db fix up scaladoc, naming of type parameters 2012-07-28 20:16:01 -07:00
Imran Rashid f1face1ea9 rename addToAccum to addAccumulator 2012-07-28 20:16:01 -07:00
Imran Rashid 2d666b9d76 add some functionality to Vector, delete copy in AccumulatorSuite 2012-07-28 20:15:51 -07:00
Imran Rashid edc6972f8e move Vector class into core and spark.util package 2012-07-28 20:15:42 -07:00
Imran Rashid 83659af11c Accumulator now inherits from Accumulable, whcih simplifies a bunch of other things (eg., no +:=) 
Conflicts:

	core/src/main/scala/spark/Accumulators.scala
 2012-07-28 20:13:51 -07:00
Imran Rashid 79d58ed20a improve scaladoc 2012-07-28 20:12:41 -07:00
Imran Rashid ae07f3864c add Accumulatable, add corresponding docs & tests for accumulators 2012-07-28 20:12:41 -07:00
Matei Zaharia 47b7ebad12 Added the Spark Streaing code, ported to Akka 2 2012-07-28 20:03:26 -07:00
Matei Zaharia dc8763fcf7 Fixed SPARK_MEM not being passed when runner is java 2012-07-28 19:53:31 -07:00
Matei Zaharia f6f917bd00 Add a sleep to prevent a failing test. 
The BlockManager's put seems to be slightly asynchronous, which can
cause it to fail this test by not removing stuff from the cache before
we put the next value. We should probably change the semantics of put()
in this case but it's hard right now. It will also be hard for
asynchronously replicated puts.
 2012-07-27 16:59:36 -07:00
Matei Zaharia c0c78d2119 Renamed test more descriptively 2012-07-27 16:28:18 -07:00
Matei Zaharia dee8ff1b9d Added a second version of union() without varargs. 2012-07-27 16:27:52 -07:00
Tathagata Das cf429699e1 Updated the new checkpoint RDD to remember partitioning of the original RDD. 2012-07-27 23:16:37 +00:00
Mosharaf Chowdhury b5be936d7c Broadcasts using BlockManager instead of BoundedMemoryCache 2012-07-27 15:38:46 -07:00
Mosharaf Chowdhury 1f19fbb8db Merge remote-tracking branch 'upstream/dev' into dev 
Conflicts:
	core/src/main/scala/spark/broadcast/Broadcast.scala
 2012-07-27 15:18:23 -07:00
Matei Zaharia b51d733a57 Fixed Java union methods having same erasure. 
Changed union() methods on lists to take a separate "first element"
argument in order to differentiate them to the compiler, because Java 7
considered it an error to have them all take Lists parameterized with
different types.
 2012-07-27 12:23:27 -07:00
Tathagata Das 3e271c3b61 Merge branch 'dev' of github.com:tdas/spark into dev 2012-07-27 12:01:04 -07:00
Tathagata Das 024905f682 Added BlockRDD and a first-cut version of checkpoint() to RDD class. 2012-07-27 12:00:49 -07:00
Tathagata Das d1eee44a03 Fixed more stuff in BoundedMemoryCache. 2012-07-27 18:33:32 +00:00
Tathagata Das d1b7f41671 Fixed bug in BoundedMemoryCache. 2012-07-27 09:00:45 -07:00
Tathagata Das 435d129bec Fixed bugs in block dropping code of MemoryStore and changed synchronized HashMap to ConcurrentHashMap in BlockManager. 2012-07-27 10:02:26 +00:00
Tathagata Das 0426769f89 Modified the block dropping code for better performance. 2012-07-26 20:53:45 -07:00
Matei Zaharia 1e2df26c33 Merge pull request #145 from squito/exp_accum 
add Accumulatable, add corresponding docs & tests for accumulators
 2012-07-26 17:25:45 -07:00
Matei Zaharia 5c5aa2ff81 Merge pull request #153 from JoshRosen/new-java-api 
Java API
 2012-07-26 17:20:52 -07:00
Josh Rosen c5e2810dc7 Add persist(), splits(), glom(), and mapPartitions() to Java API. 2012-07-26 12:46:47 -07:00
Imran Rashid 0384be3467 tasks cannot access value of accumulator 2012-07-26 12:38:51 -07:00
Josh Rosen bf61c10072 Detect non-zero exit status from PipedRDD process. 2012-07-26 11:32:59 -07:00
Josh Rosen 2a60c998cc Remove StringOps.split() from Java WordCount. 2012-07-25 10:13:06 -07:00
Josh Rosen 6a78e88237 Minor cleanup and optimizations in Java API. 
- Add override keywords.
- Cache RDDs and counts in TC example.
- Clean up JavaRDDLike's abstract methods.
 2012-07-24 09:47:00 -07:00
Denny 4f4a34c025 Stlystic changes 
Conflicts:

	core/src/test/scala/spark/MesosSchedulerSuite.scala
 2012-07-23 16:32:20 -07:00
Denny 866e6949df Always destroy SparkContext in after block for the unit tests. 
Conflicts:

	core/src/test/scala/spark/ShuffleSuite.scala
 2012-07-23 16:29:17 -07:00
Matei Zaharia 600e99728d Fix a bug where an input path was added to a Hadoop job configuration twice 2012-07-23 16:16:19 -07:00
Matei Zaharia da4298135c Merge pull request #152 from dennybritz/fix/testbeforeafter 
Always destroy SparkContext in after block for the unit tests.
 2012-07-23 16:13:24 -07:00
Denny 5656dcdfe5 Stlystic changes 2012-07-23 10:36:52 -07:00
Josh Rosen 042dcbde33 Add type annotations to Java API methods. 
Add missing Scala Map to java.util.Map conversions.
 2012-07-22 17:35:29 -07:00
Josh Rosen e23938c3be Use mapValues() in JavaPairRDD.cogroupResultToJava(). 2012-07-22 15:10:01 -07:00
Josh Rosen 460da878fc Improve Java API examples 
- Replace JavaLR example with JavaHdfsLR example.
- Use anonymous classes in JavaWordCount; add options.
- Remove @Override annotations.
 2012-07-22 14:40:39 -07:00
Matei Zaharia 840e1b21e4 Merge branch 'master' of github.com:mesos/spark 2012-07-21 21:58:40 -07:00
Matei Zaharia 6f44c0db74 Fix a bug where an input path was added to a Hadoop job configuration twice 2012-07-21 21:58:28 -07:00
Matei Zaharia d1759c0290 Merge pull request #149 from dennybritz/serfix 
Instantiating custom serializer using user's classpath
 2012-07-21 21:54:50 -07:00
Matei Zaharia 5122f11b05 Use full package name in import 2012-07-21 21:53:38 -07:00
Josh Rosen 01dce3f569 Add Java API 
Add distinct() method to RDD.

Fix bug in DoubleRDDFunctions.
 2012-07-18 17:34:29 -07:00
Denny 5559608e6f Always destroy SparkContext in after block for the unit tests. 2012-07-18 13:09:50 -07:00
Denny e4dbaf653f syntax errors 2012-07-18 12:18:00 -07:00
Denny 1d98884548 Use extended constructor in the examples. 2012-07-18 11:46:03 -07:00
Denny 2132c541f0 Create the ClassLoader before creating a SparkEnv - SparkEnv must use the loader. 2012-07-17 14:05:26 -07:00