ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
988 commits 6 branches 181 tags 397 MiB
Commit graph

988 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mosharaf Chowdhury f23395e8c5 Merge remote-tracking branch 'upstream/dev' into dev 2012-07-30 19:39:49 -07:00
Josh Rosen 509b721d12 Fix Akka configuration in assembly jar. 
This resolves an issue where running Spark from
the assembly jar would cause a "No configuration
setting found for key 'akka.version'" exception.

This solution is from the Akka Team Blog:

http://letitcrash.com/post/21025950392/
 2012-07-30 18:05:13 -07:00
Matei Zaharia 7814ecbd47 Merge remote-tracking branch 'public/dev' into dev 2012-07-30 15:05:49 -07:00
Matei Zaharia 3ee2530c0c Merge branch 'block-manager-fix' into dev 2012-07-30 13:58:46 -07:00
Matei Zaharia 400221f851 Merge branch 'dev' of git://github.com/tdas/spark into dev 2012-07-30 13:54:57 -07:00
Matei Zaharia ed1b0f8388 Made BlockManagerMaster no longer be a singleton. 
Also cleaned up a few formatting things throughout block manager code.
 2012-07-30 13:53:47 -07:00
Tathagata Das 5a26ca4a80 Restructured file locations to separate examples and other programs from core programs. 2012-07-30 13:29:13 -07:00
Matei Zaharia f471c82558 Various reorganization and formatting fixes 2012-07-30 11:24:01 -07:00
Mosharaf Chowdhury 5932a87cac Merge remote-tracking branch 'upstream/dev' into dev 2012-07-29 18:20:45 -07:00
Matei Zaharia fcee4153b9 Renamed stream package to streaming 2012-07-29 13:35:22 -07:00
Matei Zaharia e2e71a1fb5 Merge remote-tracking branch 'public/dev' into dev 2012-07-28 20:26:59 -07:00
Matei Zaharia d7f089323a Fixed AccumulatorSuite to clean up SparkContext with BeforeAndAfter 2012-07-28 20:25:42 -07:00
Imran Rashid f7149c5e46 tasks cannot access value of accumulator 2012-07-28 20:16:17 -07:00
Imran Rashid 244cbbe33a one more minor cleanup to scaladoc 2012-07-28 20:16:10 -07:00
Imran Rashid 3b392c67db fix up scaladoc, naming of type parameters 2012-07-28 20:16:01 -07:00
Imran Rashid f1face1ea9 rename addToAccum to addAccumulator 2012-07-28 20:16:01 -07:00
Imran Rashid 2d666b9d76 add some functionality to Vector, delete copy in AccumulatorSuite 2012-07-28 20:15:51 -07:00
Imran Rashid edc6972f8e move Vector class into core and spark.util package 2012-07-28 20:15:42 -07:00
Imran Rashid 83659af11c Accumulator now inherits from Accumulable, whcih simplifies a bunch of other things (eg., no +:=) 
Conflicts:

	core/src/main/scala/spark/Accumulators.scala
 2012-07-28 20:13:51 -07:00
Imran Rashid 79d58ed20a improve scaladoc 2012-07-28 20:12:41 -07:00
Imran Rashid ae07f3864c add Accumulatable, add corresponding docs & tests for accumulators 2012-07-28 20:12:41 -07:00
Matei Zaharia 47b7ebad12 Added the Spark Streaing code, ported to Akka 2 2012-07-28 20:03:26 -07:00
Matei Zaharia dc8763fcf7 Fixed SPARK_MEM not being passed when runner is java 2012-07-28 19:53:31 -07:00
Matei Zaharia f6f917bd00 Add a sleep to prevent a failing test. 
The BlockManager's put seems to be slightly asynchronous, which can
cause it to fail this test by not removing stuff from the cache before
we put the next value. We should probably change the semantics of put()
in this case but it's hard right now. It will also be hard for
asynchronously replicated puts.
 2012-07-27 16:59:36 -07:00
Matei Zaharia c0c78d2119 Renamed test more descriptively 2012-07-27 16:28:18 -07:00
Matei Zaharia dee8ff1b9d Added a second version of union() without varargs. 2012-07-27 16:27:52 -07:00
Tathagata Das cf429699e1 Updated the new checkpoint RDD to remember partitioning of the original RDD. 2012-07-27 23:16:37 +00:00
Mosharaf Chowdhury b5be936d7c Broadcasts using BlockManager instead of BoundedMemoryCache 2012-07-27 15:38:46 -07:00
Mosharaf Chowdhury 1f19fbb8db Merge remote-tracking branch 'upstream/dev' into dev 
Conflicts:
	core/src/main/scala/spark/broadcast/Broadcast.scala
 2012-07-27 15:18:23 -07:00
Matei Zaharia b51d733a57 Fixed Java union methods having same erasure. 
Changed union() methods on lists to take a separate "first element"
argument in order to differentiate them to the compiler, because Java 7
considered it an error to have them all take Lists parameterized with
different types.
 2012-07-27 12:23:27 -07:00
Tathagata Das 3e271c3b61 Merge branch 'dev' of github.com:tdas/spark into dev 2012-07-27 12:01:04 -07:00
Tathagata Das 024905f682 Added BlockRDD and a first-cut version of checkpoint() to RDD class. 2012-07-27 12:00:49 -07:00
Tathagata Das d1eee44a03 Fixed more stuff in BoundedMemoryCache. 2012-07-27 18:33:32 +00:00
Tathagata Das d1b7f41671 Fixed bug in BoundedMemoryCache. 2012-07-27 09:00:45 -07:00
Tathagata Das 435d129bec Fixed bugs in block dropping code of MemoryStore and changed synchronized HashMap to ConcurrentHashMap in BlockManager. 2012-07-27 10:02:26 +00:00
Tathagata Das 0426769f89 Modified the block dropping code for better performance. 2012-07-26 20:53:45 -07:00
Matei Zaharia 5c5aa2ff81 Merge pull request #153 from JoshRosen/new-java-api 
Java API
 2012-07-26 17:20:52 -07:00
Josh Rosen c5e2810dc7 Add persist(), splits(), glom(), and mapPartitions() to Java API. 2012-07-26 12:46:47 -07:00
Josh Rosen bf61c10072 Detect non-zero exit status from PipedRDD process. 2012-07-26 11:32:59 -07:00
Josh Rosen 2a60c998cc Remove StringOps.split() from Java WordCount. 2012-07-25 10:13:06 -07:00
Josh Rosen 6a78e88237 Minor cleanup and optimizations in Java API. 
- Add override keywords.
- Cache RDDs and counts in TC example.
- Clean up JavaRDDLike's abstract methods.
 2012-07-24 09:47:00 -07:00
Denny 4f4a34c025 Stlystic changes 
Conflicts:

	core/src/test/scala/spark/MesosSchedulerSuite.scala
 2012-07-23 16:32:20 -07:00
Denny 866e6949df Always destroy SparkContext in after block for the unit tests. 
Conflicts:

	core/src/test/scala/spark/ShuffleSuite.scala
 2012-07-23 16:29:17 -07:00
Matei Zaharia 600e99728d Fix a bug where an input path was added to a Hadoop job configuration twice 2012-07-23 16:16:19 -07:00
Josh Rosen 042dcbde33 Add type annotations to Java API methods. 
Add missing Scala Map to java.util.Map conversions.
 2012-07-22 17:35:29 -07:00
Josh Rosen e23938c3be Use mapValues() in JavaPairRDD.cogroupResultToJava(). 2012-07-22 15:10:01 -07:00
Josh Rosen 460da878fc Improve Java API examples 
- Replace JavaLR example with JavaHdfsLR example.
- Use anonymous classes in JavaWordCount; add options.
- Remove @Override annotations.
 2012-07-22 14:40:39 -07:00
Josh Rosen 01dce3f569 Add Java API 
Add distinct() method to RDD.

Fix bug in DoubleRDDFunctions.
 2012-07-18 17:34:29 -07:00
Mosharaf Chowdhury 85cd9979f2 Fix for isLocal 2012-07-13 01:13:14 -07:00
Mosharaf Chowdhury 1c83fd4b66 Merged with Upstream dev 2012-07-13 01:08:28 -07:00