ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
19847 commits 6 branches 181 tags 397 MiB
Commit graph

19847 commits

This branch
This branch
All branches
Author SHA1 Message Date
Matei Zaharia c0c78d2119 Renamed test more descriptively 2012-07-27 16:28:18 -07:00
Matei Zaharia dee8ff1b9d Added a second version of union() without varargs. 2012-07-27 16:27:52 -07:00
Tathagata Das cf429699e1 Updated the new checkpoint RDD to remember partitioning of the original RDD. 2012-07-27 23:16:37 +00:00
Mosharaf Chowdhury b5be936d7c Broadcasts using BlockManager instead of BoundedMemoryCache 2012-07-27 15:38:46 -07:00
Mosharaf Chowdhury 1f19fbb8db Merge remote-tracking branch 'upstream/dev' into dev 
Conflicts:
	core/src/main/scala/spark/broadcast/Broadcast.scala
 2012-07-27 15:18:23 -07:00
Matei Zaharia b51d733a57 Fixed Java union methods having same erasure. 
Changed union() methods on lists to take a separate "first element"
argument in order to differentiate them to the compiler, because Java 7
considered it an error to have them all take Lists parameterized with
different types.
 2012-07-27 12:23:27 -07:00
Tathagata Das 3e271c3b61 Merge branch 'dev' of github.com:tdas/spark into dev 2012-07-27 12:01:04 -07:00
Tathagata Das 024905f682 Added BlockRDD and a first-cut version of checkpoint() to RDD class. 2012-07-27 12:00:49 -07:00
Tathagata Das d1eee44a03 Fixed more stuff in BoundedMemoryCache. 2012-07-27 18:33:32 +00:00
Tathagata Das d1b7f41671 Fixed bug in BoundedMemoryCache. 2012-07-27 09:00:45 -07:00
Tathagata Das 435d129bec Fixed bugs in block dropping code of MemoryStore and changed synchronized HashMap to ConcurrentHashMap in BlockManager. 2012-07-27 10:02:26 +00:00
Tathagata Das 0426769f89 Modified the block dropping code for better performance. 2012-07-26 20:53:45 -07:00
Matei Zaharia 1e2df26c33 Merge pull request #145 from squito/exp_accum 
add Accumulatable, add corresponding docs & tests for accumulators
 2012-07-26 17:25:45 -07:00
Matei Zaharia 5c5aa2ff81 Merge pull request #153 from JoshRosen/new-java-api 
Java API
 2012-07-26 17:20:52 -07:00
Josh Rosen c5e2810dc7 Add persist(), splits(), glom(), and mapPartitions() to Java API. 2012-07-26 12:46:47 -07:00
Imran Rashid 0384be3467 tasks cannot access value of accumulator 2012-07-26 12:38:51 -07:00
Josh Rosen bf61c10072 Detect non-zero exit status from PipedRDD process. 2012-07-26 11:32:59 -07:00
Josh Rosen 2a60c998cc Remove StringOps.split() from Java WordCount. 2012-07-25 10:13:06 -07:00
Josh Rosen 6a78e88237 Minor cleanup and optimizations in Java API. 
- Add override keywords.
- Cache RDDs and counts in TC example.
- Clean up JavaRDDLike's abstract methods.
 2012-07-24 09:47:00 -07:00
Denny 4f4a34c025 Stlystic changes 
Conflicts:

	core/src/test/scala/spark/MesosSchedulerSuite.scala
 2012-07-23 16:32:20 -07:00
Denny 866e6949df Always destroy SparkContext in after block for the unit tests. 
Conflicts:

	core/src/test/scala/spark/ShuffleSuite.scala
 2012-07-23 16:29:17 -07:00
Matei Zaharia 600e99728d Fix a bug where an input path was added to a Hadoop job configuration twice 2012-07-23 16:16:19 -07:00
Matei Zaharia da4298135c Merge pull request #152 from dennybritz/fix/testbeforeafter 
Always destroy SparkContext in after block for the unit tests.
 2012-07-23 16:13:24 -07:00
Denny 5656dcdfe5 Stlystic changes 2012-07-23 10:36:52 -07:00
Josh Rosen 042dcbde33 Add type annotations to Java API methods. 
Add missing Scala Map to java.util.Map conversions.
 2012-07-22 17:35:29 -07:00
Josh Rosen e23938c3be Use mapValues() in JavaPairRDD.cogroupResultToJava(). 2012-07-22 15:10:01 -07:00
Josh Rosen 460da878fc Improve Java API examples 
- Replace JavaLR example with JavaHdfsLR example.
- Use anonymous classes in JavaWordCount; add options.
- Remove @Override annotations.
 2012-07-22 14:40:39 -07:00
Matei Zaharia 840e1b21e4 Merge branch 'master' of github.com:mesos/spark 2012-07-21 21:58:40 -07:00
Matei Zaharia 6f44c0db74 Fix a bug where an input path was added to a Hadoop job configuration twice 2012-07-21 21:58:28 -07:00
Matei Zaharia d1759c0290 Merge pull request #149 from dennybritz/serfix 
Instantiating custom serializer using user's classpath
 2012-07-21 21:54:50 -07:00
Matei Zaharia 5122f11b05 Use full package name in import 2012-07-21 21:53:38 -07:00
Josh Rosen 01dce3f569 Add Java API 
Add distinct() method to RDD.

Fix bug in DoubleRDDFunctions.
 2012-07-18 17:34:29 -07:00
Denny 5559608e6f Always destroy SparkContext in after block for the unit tests. 2012-07-18 13:09:50 -07:00
Denny e4dbaf653f syntax errors 2012-07-18 12:18:00 -07:00
Denny 1d98884548 Use extended constructor in the examples. 2012-07-18 11:46:03 -07:00
Denny 2132c541f0 Create the ClassLoader before creating a SparkEnv - SparkEnv must use the loader. 2012-07-17 14:05:26 -07:00
Denny 2b84b50a85 Use Context classloader for Serializer class 2012-07-17 13:55:23 -07:00
Imran Rashid 7f43ba7ffa one more minor cleanup to scaladoc 2012-07-16 18:26:48 -07:00
Imran Rashid 913d42c6a0 fix up scaladoc, naming of type parameters 2012-07-16 18:25:15 -07:00
Imran Rashid 85940a7d71 rename addToAccum to addAccumulator 2012-07-16 18:17:13 -07:00
Mosharaf Chowdhury 85cd9979f2 Fix for isLocal 2012-07-13 01:13:14 -07:00
Mosharaf Chowdhury 1c83fd4b66 Merged with Upstream dev 2012-07-13 01:08:28 -07:00
Mosharaf Chowdhury bb4ee580fa Cleaning BitTorrentBroadcast code... 2012-07-13 01:04:01 -07:00
Mosharaf Chowdhury 8ccffe21da Cleaned TreeBroadcast 2012-07-13 00:54:25 -07:00
Matei Zaharia a33ca6949c Merge branch 'master' of github.com:mesos/spark 2012-07-12 18:38:20 -07:00
Matei Zaharia 452330efb4 Allow null keys in Spark's reduce and group by 2012-07-12 18:37:37 -07:00
Matei Zaharia 628bb5ca7f Allow null keys in Spark's reduce and group by 2012-07-12 18:36:02 -07:00
Matei Zaharia e2a67a8024 Fixes to coarse-grained Mesos scheduler in dealing with failed nodes 2012-07-12 18:21:52 -07:00
Imran Rashid 86024ca74d add some functionality to Vector, delete copy in AccumulatorSuite 2012-07-12 13:08:03 -07:00
Imran Rashid 42ce879486 move Vector class into core and spark.util package 2012-07-12 13:08:03 -07:00