Reynold Xin
67a43bc7e6
Added a clearDependencies method in PartitionPruningRDD.
2013-01-23 23:06:52 -08:00
Reynold Xin
c109f29c97
Updated PruneDependency to change "split" to "partition".
2013-01-23 22:22:03 -08:00
Reynold Xin
eedc542a02
Removed pruneSplits method in RDD and renamed SplitsPruningRDD to
...
PartitionPruningRDD.
2013-01-23 22:14:23 -08:00
Reynold Xin
81004b967e
Marked prev RDD as transient in SplitsPruningRDD.
2013-01-23 21:54:27 -08:00
Reynold Xin
636e912f32
Created a PruneDependency to properly assign dependency for
...
SplitsPruningRDD.
2013-01-23 21:21:55 -08:00
Reynold Xin
eb222b7206
Added pruntSplits method to RDD.
2013-01-23 15:29:02 -08:00
Matei Zaharia
ebaa8f6519
Merge remote-tracking branch 'stephenh/cleanup'
...
Conflicts:
core/src/main/scala/spark/scheduler/local/LocalScheduler.scala
2013-01-22 21:05:45 -08:00
Matei Zaharia
d2d273868b
Merge pull request #397 from JoshRosen/refactoring/daemon-threads
...
Refactor daemon thread creation
2013-01-22 21:02:53 -08:00
Matei Zaharia
151c47eef5
Merge pull request #399 from NFLabs/master
...
Fix for hanging spark.HttpFileServer on the kind of virtual network
2013-01-22 15:49:24 -08:00
Stephen Haberman
27b3f3f0a9
Handle slaveLost before slaveIdToHost knows about it.
2013-01-22 15:30:42 -06:00
Stephen Haberman
588b24197a
Use default arguments instead of constructor overloads.
2013-01-22 10:19:30 -06:00
Leemoonsoo
7e9ee2e833
Fix for hanging spark.HttpFileServer with kind of virtual network
2013-01-22 23:08:34 +09:00
Josh Rosen
551a47a620
Refactor daemon thread pool creation.
2013-01-21 23:31:00 -08:00
Stephen Haberman
a8baeb9327
Further simplify getOrElse call.
2013-01-21 21:30:24 -06:00
Stephen Haberman
2d8218b871
Remove unneeded/now-broken saveAsNewAPIHadoopFile overload.
2013-01-21 20:00:27 -06:00
Stephen Haberman
ffd1623595
Minor cleanup.
2013-01-21 15:55:46 -06:00
Matei Zaharia
a88b44ed3b
Only bind to IPv4 addresses when trying to auto-detect external IP
2013-01-21 11:59:21 -08:00
Matei Zaharia
4d34c7fc3e
Fix compile error caused by cherry-pick
2013-01-21 11:33:48 -08:00
Imran Rashid
a3f571b539
more File -> String changes
2013-01-21 11:21:52 -08:00
Imran Rashid
fe26acc482
remove unused imports
2013-01-21 11:21:46 -08:00
Imran Rashid
c73107500e
send sparkHome as String instead of File over network
2013-01-21 11:21:39 -08:00
Imran Rashid
5bf73df7f0
oops, fix stupid compile error
2013-01-21 11:21:33 -08:00
Imran Rashid
aae5a920a4
get sparkHome the correct way
2013-01-21 11:21:28 -08:00
Imran Rashid
f116d6b5c6
executor can use a different sparkHome from Worker
2013-01-21 11:21:22 -08:00
Stephen Haberman
6ded481999
Merge branch 'master' into hadoopconf
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/api/java/JavaSparkContext.scala
2013-01-21 12:56:48 -06:00
Stephen Haberman
69a417858b
Also use hadoopConfiguration in newAPI methods.
2013-01-21 12:42:11 -06:00
Matei Zaharia
c0b9ceb8c3
Log remote lifecycle events in Akka for easier debugging
2013-01-21 00:23:53 -08:00
Matei Zaharia
c7b5e5f1ec
Merge pull request #389 from JoshRosen/python_rdd_checkpointing
...
Add checkpointing to the Python API
2013-01-20 17:10:44 -08:00
Josh Rosen
9f211dd3f0
Fix PythonPartitioner equality; see SPARK-654.
...
PythonPartitioner did not take the Python-side partitioning function
into account when checking for equality, which might cause problems
in the future.
2013-01-20 15:41:42 -08:00
Josh Rosen
5b6ea9e9a0
Update checkpointing API docs in Python/Java.
2013-01-20 15:31:41 -08:00
Josh Rosen
7ed1bf4b48
Add RDD checkpointing to Python API.
2013-01-20 13:19:19 -08:00
Matei Zaharia
86057ec7c8
Merge branch 'master' into streaming
...
Conflicts:
core/src/main/scala/spark/api/python/PythonRDD.scala
2013-01-20 12:47:55 -08:00
Matei Zaharia
8e7f098a2c
Added accumulators to PySpark
2013-01-20 01:57:44 -08:00
Tathagata Das
4f8fe58b25
Merge branch 'mesos-streaming' into streaming
...
Conflicts:
core/src/main/scala/spark/api/java/JavaRDDLike.scala
core/src/main/scala/spark/api/java/JavaSparkContext.scala
core/src/test/scala/spark/JavaAPISuite.java
2013-01-20 01:13:56 -08:00
Tathagata Das
214345ceac
Fixed issue https://spark-project.atlassian.net/browse/STREAMING-29 , along with updates to doc comments in SparkContext.checkpoint().
2013-01-19 23:50:17 -08:00
Patrick Wendell
ee0314c3b3
Merge branch 'streaming' into streaming-java-api
2013-01-17 18:43:00 -08:00
Patrick Wendell
d5570c7968
Adding checkpointing to Java API
2013-01-17 18:41:58 -08:00
Matei Zaharia
54c0f9f185
Fix code that assumed spark.local.dir is only a single directory
2013-01-17 17:40:55 -08:00
Fernand Pajot
742bc841ad
changed HttpBroadcast server cache to be in spark.local.dir instead of java.io.tmpdir
2013-01-17 16:56:11 -08:00
Matei Zaharia
aff1844155
Merge pull request #381 from squito/remove_threadpool
...
remove unused thread pool
2013-01-16 16:46:42 -08:00
Tathagata Das
f466ee44bc
Merge branch 'master' into streaming
...
Conflicts:
core/src/main/scala/spark/MapOutputTracker.scala
2013-01-16 12:57:11 -08:00
Imran Rashid
eae698f755
remove unused thread pool
2013-01-16 12:21:37 -08:00
Tathagata Das
a805ac4a7c
Disabled checkpoint for PairwiseRDD (pySpark).
2013-01-16 10:55:26 -08:00
Matei Zaharia
4beb084f64
Merge pull request #374 from woggling/null-mapout
...
Generate FetchFailedException even for cached missing map outputs
2013-01-15 14:22:29 -08:00
Tathagata Das
cd1521cfdb
Merge branch 'master' into streaming
...
Conflicts:
core/src/main/scala/spark/rdd/CoGroupedRDD.scala
core/src/main/scala/spark/rdd/FilteredRDD.scala
docs/_layouts/global.html
docs/index.md
run
2013-01-15 12:08:51 -08:00
Stephen Haberman
dd583b7ebf
Call executeOnCompleteCallbacks in a finally block.
2013-01-15 10:52:06 -06:00
Tathagata Das
eded21925a
Merge pull request #375 from tdas/streaming
...
Important bug fixes
2013-01-14 23:06:40 -08:00
Charles Reiss
273fb5cc10
Throw FetchFailedException for cached missing locs
2013-01-14 15:26:48 -08:00
Tathagata Das
131be5d62e
Fixed bug in RDD checkpointing.
2013-01-14 03:28:25 -08:00
Tathagata Das
82b0cc90ca
Merge pull request #370 from tdas/streaming
...
Added more documentation and minor change in API for NetworkReceiver
2013-01-13 21:28:12 -08:00