Matei Zaharia
545165e815
Merge pull request #158 from JoshRosen/fix/assembly-akka-config
...
Merge Akka reference.conf files in sbt assembly task
2012-07-30 22:37:00 -07:00
Mosharaf Chowdhury
f23395e8c5
Merge remote-tracking branch 'upstream/dev' into dev
2012-07-30 19:39:49 -07:00
Josh Rosen
509b721d12
Fix Akka configuration in assembly jar.
...
This resolves an issue where running Spark from
the assembly jar would cause a "No configuration
setting found for key 'akka.version'" exception.
This solution is from the Akka Team Blog:
http://letitcrash.com/post/21025950392/
2012-07-30 18:05:13 -07:00
Matei Zaharia
7814ecbd47
Merge remote-tracking branch 'public/dev' into dev
2012-07-30 15:05:49 -07:00
Matei Zaharia
3ee2530c0c
Merge branch 'block-manager-fix' into dev
2012-07-30 13:58:46 -07:00
Matei Zaharia
400221f851
Merge branch 'dev' of git://github.com/tdas/spark into dev
2012-07-30 13:54:57 -07:00
Matei Zaharia
ed1b0f8388
Made BlockManagerMaster no longer be a singleton.
...
Also cleaned up a few formatting things throughout block manager code.
2012-07-30 13:53:47 -07:00
Tathagata Das
5a26ca4a80
Restructured file locations to separate examples and other programs from core programs.
2012-07-30 13:29:13 -07:00
Matei Zaharia
f471c82558
Various reorganization and formatting fixes
2012-07-30 11:24:01 -07:00
Paul Cavallaro
e3952f31de
Logging Throwables in Info and Debug
...
Logging Throwables in logInfo and logDebug instead of swallowing them.
2012-07-30 13:41:09 -04:00
Mosharaf Chowdhury
5932a87cac
Merge remote-tracking branch 'upstream/dev' into dev
2012-07-29 18:20:45 -07:00
Matei Zaharia
fcee4153b9
Renamed stream package to streaming
2012-07-29 13:35:22 -07:00
Matei Zaharia
e2e71a1fb5
Merge remote-tracking branch 'public/dev' into dev
2012-07-28 20:26:59 -07:00
Matei Zaharia
d7f089323a
Fixed AccumulatorSuite to clean up SparkContext with BeforeAndAfter
2012-07-28 20:25:42 -07:00
Imran Rashid
f7149c5e46
tasks cannot access value of accumulator
2012-07-28 20:16:17 -07:00
Imran Rashid
244cbbe33a
one more minor cleanup to scaladoc
2012-07-28 20:16:10 -07:00
Imran Rashid
3b392c67db
fix up scaladoc, naming of type parameters
2012-07-28 20:16:01 -07:00
Imran Rashid
f1face1ea9
rename addToAccum to addAccumulator
2012-07-28 20:16:01 -07:00
Imran Rashid
2d666b9d76
add some functionality to Vector, delete copy in AccumulatorSuite
2012-07-28 20:15:51 -07:00
Imran Rashid
edc6972f8e
move Vector class into core and spark.util package
2012-07-28 20:15:42 -07:00
Imran Rashid
83659af11c
Accumulator now inherits from Accumulable, whcih simplifies a bunch of other things (eg., no +:=)
...
Conflicts:
core/src/main/scala/spark/Accumulators.scala
2012-07-28 20:13:51 -07:00
Imran Rashid
79d58ed20a
improve scaladoc
2012-07-28 20:12:41 -07:00
Imran Rashid
ae07f3864c
add Accumulatable, add corresponding docs & tests for accumulators
2012-07-28 20:12:41 -07:00
Matei Zaharia
47b7ebad12
Added the Spark Streaing code, ported to Akka 2
2012-07-28 20:03:26 -07:00
Matei Zaharia
dc8763fcf7
Fixed SPARK_MEM not being passed when runner is java
2012-07-28 19:53:31 -07:00
Matei Zaharia
f6f917bd00
Add a sleep to prevent a failing test.
...
The BlockManager's put seems to be slightly asynchronous, which can
cause it to fail this test by not removing stuff from the cache before
we put the next value. We should probably change the semantics of put()
in this case but it's hard right now. It will also be hard for
asynchronously replicated puts.
2012-07-27 16:59:36 -07:00
Matei Zaharia
c0c78d2119
Renamed test more descriptively
2012-07-27 16:28:18 -07:00
Matei Zaharia
dee8ff1b9d
Added a second version of union() without varargs.
2012-07-27 16:27:52 -07:00
Tathagata Das
cf429699e1
Updated the new checkpoint RDD to remember partitioning of the original RDD.
2012-07-27 23:16:37 +00:00
Mosharaf Chowdhury
b5be936d7c
Broadcasts using BlockManager instead of BoundedMemoryCache
2012-07-27 15:38:46 -07:00
Mosharaf Chowdhury
1f19fbb8db
Merge remote-tracking branch 'upstream/dev' into dev
...
Conflicts:
core/src/main/scala/spark/broadcast/Broadcast.scala
2012-07-27 15:18:23 -07:00
Matei Zaharia
b51d733a57
Fixed Java union methods having same erasure.
...
Changed union() methods on lists to take a separate "first element"
argument in order to differentiate them to the compiler, because Java 7
considered it an error to have them all take Lists parameterized with
different types.
2012-07-27 12:23:27 -07:00
Tathagata Das
3e271c3b61
Merge branch 'dev' of github.com:tdas/spark into dev
2012-07-27 12:01:04 -07:00
Tathagata Das
024905f682
Added BlockRDD and a first-cut version of checkpoint() to RDD class.
2012-07-27 12:00:49 -07:00
Tathagata Das
d1eee44a03
Fixed more stuff in BoundedMemoryCache.
2012-07-27 18:33:32 +00:00
Tathagata Das
d1b7f41671
Fixed bug in BoundedMemoryCache.
2012-07-27 09:00:45 -07:00
Tathagata Das
435d129bec
Fixed bugs in block dropping code of MemoryStore and changed synchronized HashMap to ConcurrentHashMap in BlockManager.
2012-07-27 10:02:26 +00:00
Tathagata Das
0426769f89
Modified the block dropping code for better performance.
2012-07-26 20:53:45 -07:00
Matei Zaharia
1e2df26c33
Merge pull request #145 from squito/exp_accum
...
add Accumulatable, add corresponding docs & tests for accumulators
2012-07-26 17:25:45 -07:00
Matei Zaharia
5c5aa2ff81
Merge pull request #153 from JoshRosen/new-java-api
...
Java API
2012-07-26 17:20:52 -07:00
Josh Rosen
c5e2810dc7
Add persist(), splits(), glom(), and mapPartitions() to Java API.
2012-07-26 12:46:47 -07:00
Imran Rashid
0384be3467
tasks cannot access value of accumulator
2012-07-26 12:38:51 -07:00
Josh Rosen
bf61c10072
Detect non-zero exit status from PipedRDD process.
2012-07-26 11:32:59 -07:00
Josh Rosen
2a60c998cc
Remove StringOps.split() from Java WordCount.
2012-07-25 10:13:06 -07:00
Josh Rosen
6a78e88237
Minor cleanup and optimizations in Java API.
...
- Add override keywords.
- Cache RDDs and counts in TC example.
- Clean up JavaRDDLike's abstract methods.
2012-07-24 09:47:00 -07:00
Denny
4f4a34c025
Stlystic changes
...
Conflicts:
core/src/test/scala/spark/MesosSchedulerSuite.scala
2012-07-23 16:32:20 -07:00
Denny
866e6949df
Always destroy SparkContext in after block for the unit tests.
...
Conflicts:
core/src/test/scala/spark/ShuffleSuite.scala
2012-07-23 16:29:17 -07:00
Matei Zaharia
600e99728d
Fix a bug where an input path was added to a Hadoop job configuration twice
2012-07-23 16:16:19 -07:00
Matei Zaharia
da4298135c
Merge pull request #152 from dennybritz/fix/testbeforeafter
...
Always destroy SparkContext in after block for the unit tests.
2012-07-23 16:13:24 -07:00
Denny
5656dcdfe5
Stlystic changes
2012-07-23 10:36:52 -07:00