Tathagata Das
cf429699e1
Updated the new checkpoint RDD to remember partitioning of the original RDD.
2012-07-27 23:16:37 +00:00
Mosharaf Chowdhury
b5be936d7c
Broadcasts using BlockManager instead of BoundedMemoryCache
2012-07-27 15:38:46 -07:00
Mosharaf Chowdhury
1f19fbb8db
Merge remote-tracking branch 'upstream/dev' into dev
...
Conflicts:
core/src/main/scala/spark/broadcast/Broadcast.scala
2012-07-27 15:18:23 -07:00
Matei Zaharia
b51d733a57
Fixed Java union methods having same erasure.
...
Changed union() methods on lists to take a separate "first element"
argument in order to differentiate them to the compiler, because Java 7
considered it an error to have them all take Lists parameterized with
different types.
2012-07-27 12:23:27 -07:00
Tathagata Das
3e271c3b61
Merge branch 'dev' of github.com:tdas/spark into dev
2012-07-27 12:01:04 -07:00
Tathagata Das
024905f682
Added BlockRDD and a first-cut version of checkpoint() to RDD class.
2012-07-27 12:00:49 -07:00
Tathagata Das
d1eee44a03
Fixed more stuff in BoundedMemoryCache.
2012-07-27 18:33:32 +00:00
Tathagata Das
d1b7f41671
Fixed bug in BoundedMemoryCache.
2012-07-27 09:00:45 -07:00
Tathagata Das
435d129bec
Fixed bugs in block dropping code of MemoryStore and changed synchronized HashMap to ConcurrentHashMap in BlockManager.
2012-07-27 10:02:26 +00:00
Tathagata Das
0426769f89
Modified the block dropping code for better performance.
2012-07-26 20:53:45 -07:00
Matei Zaharia
1e2df26c33
Merge pull request #145 from squito/exp_accum
...
add Accumulatable, add corresponding docs & tests for accumulators
2012-07-26 17:25:45 -07:00
Matei Zaharia
5c5aa2ff81
Merge pull request #153 from JoshRosen/new-java-api
...
Java API
2012-07-26 17:20:52 -07:00
Josh Rosen
c5e2810dc7
Add persist(), splits(), glom(), and mapPartitions() to Java API.
2012-07-26 12:46:47 -07:00
Imran Rashid
0384be3467
tasks cannot access value of accumulator
2012-07-26 12:38:51 -07:00
Josh Rosen
bf61c10072
Detect non-zero exit status from PipedRDD process.
2012-07-26 11:32:59 -07:00
Josh Rosen
2a60c998cc
Remove StringOps.split() from Java WordCount.
2012-07-25 10:13:06 -07:00
Josh Rosen
6a78e88237
Minor cleanup and optimizations in Java API.
...
- Add override keywords.
- Cache RDDs and counts in TC example.
- Clean up JavaRDDLike's abstract methods.
2012-07-24 09:47:00 -07:00
Denny
4f4a34c025
Stlystic changes
...
Conflicts:
core/src/test/scala/spark/MesosSchedulerSuite.scala
2012-07-23 16:32:20 -07:00
Denny
866e6949df
Always destroy SparkContext in after block for the unit tests.
...
Conflicts:
core/src/test/scala/spark/ShuffleSuite.scala
2012-07-23 16:29:17 -07:00
Matei Zaharia
600e99728d
Fix a bug where an input path was added to a Hadoop job configuration twice
2012-07-23 16:16:19 -07:00
Matei Zaharia
da4298135c
Merge pull request #152 from dennybritz/fix/testbeforeafter
...
Always destroy SparkContext in after block for the unit tests.
2012-07-23 16:13:24 -07:00
Denny
5656dcdfe5
Stlystic changes
2012-07-23 10:36:52 -07:00
Josh Rosen
042dcbde33
Add type annotations to Java API methods.
...
Add missing Scala Map to java.util.Map conversions.
2012-07-22 17:35:29 -07:00
Josh Rosen
e23938c3be
Use mapValues() in JavaPairRDD.cogroupResultToJava().
2012-07-22 15:10:01 -07:00
Josh Rosen
460da878fc
Improve Java API examples
...
- Replace JavaLR example with JavaHdfsLR example.
- Use anonymous classes in JavaWordCount; add options.
- Remove @Override annotations.
2012-07-22 14:40:39 -07:00
Matei Zaharia
840e1b21e4
Merge branch 'master' of github.com:mesos/spark
2012-07-21 21:58:40 -07:00
Matei Zaharia
6f44c0db74
Fix a bug where an input path was added to a Hadoop job configuration twice
2012-07-21 21:58:28 -07:00
Matei Zaharia
d1759c0290
Merge pull request #149 from dennybritz/serfix
...
Instantiating custom serializer using user's classpath
2012-07-21 21:54:50 -07:00
Matei Zaharia
5122f11b05
Use full package name in import
2012-07-21 21:53:38 -07:00
Josh Rosen
01dce3f569
Add Java API
...
Add distinct() method to RDD.
Fix bug in DoubleRDDFunctions.
2012-07-18 17:34:29 -07:00
Denny
5559608e6f
Always destroy SparkContext in after block for the unit tests.
2012-07-18 13:09:50 -07:00
Denny
e4dbaf653f
syntax errors
2012-07-18 12:18:00 -07:00
Denny
1d98884548
Use extended constructor in the examples.
2012-07-18 11:46:03 -07:00
Denny
2132c541f0
Create the ClassLoader before creating a SparkEnv - SparkEnv must use the loader.
2012-07-17 14:05:26 -07:00
Denny
2b84b50a85
Use Context classloader for Serializer class
2012-07-17 13:55:23 -07:00
Imran Rashid
7f43ba7ffa
one more minor cleanup to scaladoc
2012-07-16 18:26:48 -07:00
Imran Rashid
913d42c6a0
fix up scaladoc, naming of type parameters
2012-07-16 18:25:15 -07:00
Imran Rashid
85940a7d71
rename addToAccum to addAccumulator
2012-07-16 18:17:13 -07:00
Mosharaf Chowdhury
85cd9979f2
Fix for isLocal
2012-07-13 01:13:14 -07:00
Mosharaf Chowdhury
1c83fd4b66
Merged with Upstream dev
2012-07-13 01:08:28 -07:00
Mosharaf Chowdhury
bb4ee580fa
Cleaning BitTorrentBroadcast code...
2012-07-13 01:04:01 -07:00
Mosharaf Chowdhury
8ccffe21da
Cleaned TreeBroadcast
2012-07-13 00:54:25 -07:00
Matei Zaharia
a33ca6949c
Merge branch 'master' of github.com:mesos/spark
2012-07-12 18:38:20 -07:00
Matei Zaharia
452330efb4
Allow null keys in Spark's reduce and group by
2012-07-12 18:37:37 -07:00
Matei Zaharia
628bb5ca7f
Allow null keys in Spark's reduce and group by
2012-07-12 18:36:02 -07:00
Matei Zaharia
e2a67a8024
Fixes to coarse-grained Mesos scheduler in dealing with failed nodes
2012-07-12 18:21:52 -07:00
Imran Rashid
86024ca74d
add some functionality to Vector, delete copy in AccumulatorSuite
2012-07-12 13:08:03 -07:00
Imran Rashid
42ce879486
move Vector class into core and spark.util package
2012-07-12 13:08:03 -07:00
Imran Rashid
13cc72cfb5
Accumulator now inherits from Accumulable, whcih simplifies a bunch of other things (eg., no +:=)
2012-07-12 12:40:10 -07:00
Imran Rashid
73935629a1
improve scaladoc
2012-07-12 09:58:06 -07:00