Tathagata Das
dff53d1b94
Merge branch 'mesos-master' into streaming
2013-02-24 12:17:22 -08:00
Matei Zaharia
3b9f929467
Merge pull request #468 from haitaoyao/master
...
support customized java options for Master, Worker, Executor, and Repl
2013-02-23 23:38:15 -08:00
Stephen Haberman
37c7a71f9c
Add subtract to JavaRDD, JavaDoubleRDD, and JavaPairRDD.
2013-02-24 00:27:53 -06:00
Stephen Haberman
f442e7d83c
Update for split->partition rename.
2013-02-24 00:27:14 -06:00
Stephen Haberman
cec87a0653
Merge branch 'master' into subtract
2013-02-23 23:27:55 -06:00
Tathagata Das
d853aa9658
Change spark.cleaner.delay to spark.cleaner.ttl. Updated docs.
2013-02-23 17:42:26 -08:00
Patrick Wendell
931f439be9
Responding to code review
2013-02-23 15:40:41 -08:00
Patrick Wendell
f51b0f93f2
Adding Java-accessible methods to Vector.scala
...
This is needed for the Strata machine learning tutorial (and
also is generally helpful).
2013-02-23 13:26:59 -08:00
Matei Zaharia
d942d39072
Handle exceptions in RecordReader.close() better (suggested by Jim
...
Donahue)
2013-02-23 11:19:07 -08:00
Matei Zaharia
c89824046a
Merge pull request #490 from woggling/conn-death
...
Detect when SendingConnections disconnect even if we aren't sending to them
2013-02-22 22:58:19 -08:00
Charles Reiss
50cf8c8b79
Add fault tolerance test that uses replicated RDDs.
2013-02-22 16:11:53 -08:00
Charles Reiss
c8a7886921
Detect when SendingConnections drop by trying to read them.
...
Comment fix
2013-02-22 16:11:52 -08:00
Matei Zaharia
d4d7993bf5
Several fixes to the work to log when no resources can be used by a job.
...
Fixed some of the messages as well as code style.
2013-02-22 15:51:37 -08:00
Matei Zaharia
f33662c133
Merge remote-tracking branch 'pwendell/starvation-check'
...
Also fixed a bug where master was offering executors on dead workers
Conflicts:
core/src/main/scala/spark/deploy/master/Master.scala
2013-02-22 15:27:41 -08:00
Matei Zaharia
7341de0d48
Merge pull request #475 from JoshRosen/spark-668
...
Remove hack workaround for SPARK-668
2013-02-22 14:56:18 -08:00
Patrick Wendell
f8c3a03d55
SPARK-702: Replace Function --> JFunction in JavaAPI Suite.
...
In a few places the Scala (rather than Java) function class is used.
2013-02-22 12:54:15 -08:00
haitao.yao
8215b95547
Merge branch 'mesos'
2013-02-21 10:07:24 +08:00
Tathagata Das
334ab92441
Fixed bug in CheckpointSuite
2013-02-20 10:26:36 -08:00
Tathagata Das
1cb725e417
Merge branch 'mesos-master' into streaming
2013-02-20 09:55:35 -08:00
Tathagata Das
fb9956256d
Merge branch 'mesos-master' into streaming
...
Conflicts:
core/src/main/scala/spark/rdd/CheckpointRDD.scala
streaming/src/main/scala/spark/streaming/dstream/ReducedWindowedDStream.scala
2013-02-20 09:01:29 -08:00
Matei Zaharia
05bc02e80b
Merge pull request #482 from woggling/shutdown-exceptions
...
Don't call System.exit over uncaught exceptions from shutdown hooks
2013-02-19 20:56:15 -08:00
haitao.yao
6a3d44c673
Merge branch 'mesos'
2013-02-20 10:23:58 +08:00
Charles Reiss
092c631fa8
Pull detection of being in a shutdown hook into utility function.
2013-02-19 17:49:55 -08:00
Reynold Xin
130f704baf
Added a method to create PartitionPruningRDD.
2013-02-19 16:03:52 -08:00
Charles Reiss
d0588bd6d7
Catch/log errors deleting temp dirs
2013-02-19 13:04:06 -08:00
Charles Reiss
687581c3ec
Paranoid uncaught exception handling for exceptions during shutdown
2013-02-19 13:03:02 -08:00
haitao.yao
7c129388fb
Merge branch 'mesos'
2013-02-19 11:22:24 +08:00
Matei Zaharia
7151e1e4c8
Rename "jobs" to "applications" in the standalone cluster
2013-02-17 23:23:08 -08:00
Matei Zaharia
06e5e6627f
Renamed "splits" to "partitions"
2013-02-17 22:13:26 -08:00
Matei Zaharia
340cc54e47
Merge pull request #471 from stephenh/parallelrdd
...
Move ParallelCollection into spark.rdd package.
2013-02-16 16:39:15 -08:00
Matei Zaharia
3260b6120e
Merge pull request #470 from stephenh/morek
...
Make CoGroupedRDDs explicitly have the same key type.
2013-02-16 16:38:38 -08:00
Stephen Haberman
924f47dd11
Add RDD.subtract.
...
Instead of reusing the cogroup primitive, this adds a SubtractedRDD
that knows it only needs to keep rdd1's values (per split) in memory.
2013-02-16 13:38:42 -06:00
Stephen Haberman
e7713adb99
Move ParallelCollection into spark.rdd package.
2013-02-16 13:20:48 -06:00
Stephen Haberman
ae2234687d
Make CoGroupedRDDs explicitly have the same key type.
2013-02-16 13:10:31 -06:00
Stephen Haberman
4328873294
Add assertion about dependencies.
2013-02-16 01:16:40 -06:00
Stephen Haberman
c34b8ad2c5
Avoid a shuffle if combineByKey is passed the same partitioner.
2013-02-16 00:54:03 -06:00
haitao.yao
a9cfac347a
Merge branch 'mesos'
2013-02-16 10:11:28 +08:00
Imran Rashid
893bad9089
use appid instead of frameworkid; simplify stupid condition
2013-02-13 20:30:21 -08:00
Imran Rashid
8f18e7e863
include jobid in Executor commandline args
2013-02-13 13:05:13 -08:00
Matei Zaharia
fd7e414bd0
Merge pull request #464 from pwendell/java-type-fix
...
SPARK-694: All references to [K, V] in JavaDStreamLike should be changed to [K2, V2]
2013-02-11 19:19:05 -08:00
Matei Zaharia
bfeed4725d
Merge pull request #465 from pwendell/java-sort-fix
...
SPARK-696: sortByKey should use 'ascending' parameter
2013-02-11 18:23:12 -08:00
Patrick Wendell
21df6ffc13
SPARK-696: sortByKey should use 'ascending' parameter
2013-02-11 17:43:26 -08:00
Matei Zaharia
ea08537143
Fixed an exponential recursion that could happen with doCheckpoint due
...
to lack of memoization
2013-02-11 13:23:50 -08:00
Josh Rosen
e9fb25426e
Remove hack workaround for SPARK-668.
...
Renaming the type paramters solves this problem (see SPARK-694).
I tried this fix earlier, but it didn't work because I didn't run
`sbt/sbt clean` first.
2013-02-11 11:19:20 -08:00
Patrick Wendell
f0b68c623c
Initial cut at replacing K, V in Java files
2013-02-11 10:03:37 -08:00
Matei Zaharia
da8afbc77e
Some bug and formatting fixes to FT
...
Conflicts:
core/src/main/scala/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
2013-02-10 22:43:38 -08:00
root
1b47fa2752
Detect hard crashes of workers using a heartbeat mechanism.
...
Also fixes some issues in the rest of the code with detecting workers this way.
Conflicts:
core/src/main/scala/spark/deploy/master/Master.scala
core/src/main/scala/spark/deploy/worker/Worker.scala
core/src/main/scala/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneClusterMessage.scala
core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
2013-02-10 22:28:28 -08:00
Matei Zaharia
8c66c49962
Tweak web UI so that people don't get confused about master URL format
...
Conflicts:
core/src/main/twirl/spark/deploy/master/index.scala.html
core/src/main/twirl/spark/deploy/worker/index.scala.html
2013-02-10 21:58:34 -08:00
Tathagata Das
16baea62bc
Fixed bug in CheckpointRDD to prevent exception when the original RDD had zero splits.
2013-02-10 19:14:49 -08:00
Patrick Wendell
2ed791fd7f
Minor fixes
2013-02-09 22:00:38 -08:00