Matei Zaharia
151c47eef5
Merge pull request #399 from NFLabs/master
...
Fix for hanging spark.HttpFileServer on the kind of virtual network
2013-01-22 15:49:24 -08:00
Matei Zaharia
49c78446c8
Merge pull request #402 from stephenh/handleslavelosttoosoon
...
Handle slaveLost before slaveIdToHost knows about it.
2013-01-22 15:48:37 -08:00
Stephen Haberman
27b3f3f0a9
Handle slaveLost before slaveIdToHost knows about it.
2013-01-22 15:30:42 -06:00
Matei Zaharia
d1de9d7d15
Merge pull request #400 from stephenh/sparkcontextcstrs
...
Use default arguments instead of constructor overloads.
2013-01-22 11:56:54 -08:00
Stephen Haberman
588b24197a
Use default arguments instead of constructor overloads.
2013-01-22 10:19:30 -06:00
Leemoonsoo
7e9ee2e833
Fix for hanging spark.HttpFileServer with kind of virtual network
2013-01-22 23:08:34 +09:00
Charles Reiss
e353886a8c
Use generation numbers for fetch failure tracking
2013-01-22 00:23:31 -08:00
Josh Rosen
551a47a620
Refactor daemon thread pool creation.
2013-01-21 23:31:00 -08:00
Josh Rosen
c75ae3622e
Make AccumulatorParam an abstract base class.
2013-01-21 22:32:57 -08:00
Stephen Haberman
a8baeb9327
Further simplify getOrElse call.
2013-01-21 21:30:24 -06:00
Matei Zaharia
7d3e359f2c
Merge pull request #395 from stephenh/fixrecursion
...
Remove unneeded/now-broken saveAsNewAPIHadoopFile overload.
2013-01-21 18:11:31 -08:00
Stephen Haberman
2d8218b871
Remove unneeded/now-broken saveAsNewAPIHadoopFile overload.
2013-01-21 20:00:27 -06:00
Josh Rosen
7b9e96c992
Add synchronization to Executor.updateDependencies() (SPARK-662)
2013-01-21 17:34:23 -08:00
Josh Rosen
ef711902c1
Don't download files to master's working directory.
...
This should avoid exceptions caused by existing
files with different contents.
I also removed some unused code.
2013-01-21 17:34:17 -08:00
Matei Zaharia
506077c993
Merge pull request #392 from stephenh/javaapisuite
...
Move JavaAPISuite into spark.streaming.
2013-01-21 15:07:43 -08:00
Stephen Haberman
e5ca241335
Move JavaAPISuite into spark.streaming.
2013-01-21 16:06:58 -06:00
Stephen Haberman
ffd1623595
Minor cleanup.
2013-01-21 15:55:46 -06:00
Matei Zaharia
76d7c0ce2b
Add more Akka settings to docs
2013-01-21 13:10:33 -08:00
Matei Zaharia
2173f6c7ca
Clarify the documentation on env variables for standalone mode
2013-01-21 13:03:20 -08:00
Matei Zaharia
a88b44ed3b
Only bind to IPv4 addresses when trying to auto-detect external IP
2013-01-21 11:59:21 -08:00
Matei Zaharia
4d34c7fc3e
Fix compile error caused by cherry-pick
2013-01-21 11:33:48 -08:00
Imran Rashid
a3f571b539
more File -> String changes
2013-01-21 11:21:52 -08:00
Imran Rashid
fe26acc482
remove unused imports
2013-01-21 11:21:46 -08:00
Imran Rashid
c73107500e
send sparkHome as String instead of File over network
2013-01-21 11:21:39 -08:00
Imran Rashid
5bf73df7f0
oops, fix stupid compile error
2013-01-21 11:21:33 -08:00
Imran Rashid
aae5a920a4
get sparkHome the correct way
2013-01-21 11:21:28 -08:00
Imran Rashid
f116d6b5c6
executor can use a different sparkHome from Worker
2013-01-21 11:21:22 -08:00
Matei Zaharia
414b41e1d8
Merge pull request #362 from stephenh/hadoopconf
...
Add SparkContext.hadoopConfiguration
2013-01-21 11:18:48 -08:00
Stephen Haberman
6ded481999
Merge branch 'master' into hadoopconf
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
core/src/main/scala/spark/api/java/JavaSparkContext.scala
2013-01-21 12:56:48 -06:00
Stephen Haberman
69a417858b
Also use hadoopConfiguration in newAPI methods.
2013-01-21 12:42:11 -06:00
Matei Zaharia
c0b9ceb8c3
Log remote lifecycle events in Akka for easier debugging
2013-01-21 00:23:53 -08:00
Matei Zaharia
4750907c3d
Update run script to deal with change to build of REPL shaded JAR
2013-01-20 21:05:17 -08:00
Matei Zaharia
6e3754bf47
Add Maven build file for streaming, and fix some issues in SBT file
...
As part of this, changed our Scala 2.9.2 Kafka library to be available
as a local Maven repository, following the example in
(http://blog.dub.podval.org/2010/01/maven-in-project-repository.html )
2013-01-20 19:22:24 -08:00
Matei Zaharia
c7b5e5f1ec
Merge pull request #389 from JoshRosen/python_rdd_checkpointing
...
Add checkpointing to the Python API
2013-01-20 17:10:44 -08:00
Matei Zaharia
14360aba89
Merge pull request #390 from JoshRosen/spark-654
...
Fix PythonPartitioner equality
2013-01-20 17:08:53 -08:00
Josh Rosen
9f211dd3f0
Fix PythonPartitioner equality; see SPARK-654.
...
PythonPartitioner did not take the Python-side partitioning function
into account when checking for equality, which might cause problems
in the future.
2013-01-20 15:41:42 -08:00
Josh Rosen
00d70cd660
Clean up setup code in PySpark checkpointing tests
2013-01-20 15:38:11 -08:00
Josh Rosen
5b6ea9e9a0
Update checkpointing API docs in Python/Java.
2013-01-20 15:31:41 -08:00
Josh Rosen
d0ba80dc72
Add checkpointFile() and more tests to PySpark.
2013-01-20 13:59:45 -08:00
Josh Rosen
7ed1bf4b48
Add RDD checkpointing to Python API.
2013-01-20 13:19:19 -08:00
Matei Zaharia
fe85a07511
Merge pull request #361 from mesos/streaming
...
Merge Streaming into master
2013-01-20 12:48:15 -08:00
Matei Zaharia
86057ec7c8
Merge branch 'master' into streaming
...
Conflicts:
core/src/main/scala/spark/api/python/PythonRDD.scala
2013-01-20 12:47:55 -08:00
Josh Rosen
17035db159
Add __repr__ to Accumulator; fix bug in sc.accumulator
2013-01-20 11:58:57 -08:00
Josh Rosen
9f54d7e1f5
Merge pull request #387 from mateiz/python-accumulators
...
Add accumulators to PySpark
2013-01-20 11:00:36 -08:00
Matei Zaharia
2a8c2a6790
Minor formatting fixes
2013-01-20 10:24:53 -08:00
Matei Zaharia
5d473f050e
Merge pull request #376 from MLnick/python-als
...
Python ALS example
2013-01-20 10:21:29 -08:00
Matei Zaharia
922c5ec069
Merge pull request #385 from pwendell/ec2-guide-fix
...
Clarifying log directory in EC2 guide
2013-01-20 10:05:38 -08:00
Patrick Wendell
5f74ead636
Changes based on Matei's comment
2013-01-20 08:59:20 -08:00
Tathagata Das
76ff962edc
Merge pull request #380 from tdas/streaming
...
Merging pySpark to streaming
2013-01-20 03:54:46 -08:00
Tathagata Das
33bad85bb9
Fixed streaming testsuite bugs
2013-01-20 03:51:11 -08:00