Josh Rosen
9f211dd3f0
Fix PythonPartitioner equality; see SPARK-654.
...
PythonPartitioner did not take the Python-side partitioning function
into account when checking for equality, which might cause problems
in the future.
2013-01-20 15:41:42 -08:00
Josh Rosen
00d70cd660
Clean up setup code in PySpark checkpointing tests
2013-01-20 15:38:11 -08:00
Josh Rosen
5b6ea9e9a0
Update checkpointing API docs in Python/Java.
2013-01-20 15:31:41 -08:00
Josh Rosen
d0ba80dc72
Add checkpointFile() and more tests to PySpark.
2013-01-20 13:59:45 -08:00
Josh Rosen
7ed1bf4b48
Add RDD checkpointing to Python API.
2013-01-20 13:19:19 -08:00
Matei Zaharia
fe85a07511
Merge pull request #361 from mesos/streaming
...
Merge Streaming into master
2013-01-20 12:48:15 -08:00
Matei Zaharia
86057ec7c8
Merge branch 'master' into streaming
...
Conflicts:
core/src/main/scala/spark/api/python/PythonRDD.scala
2013-01-20 12:47:55 -08:00
Josh Rosen
17035db159
Add __repr__ to Accumulator; fix bug in sc.accumulator
2013-01-20 11:58:57 -08:00
Josh Rosen
9f54d7e1f5
Merge pull request #387 from mateiz/python-accumulators
...
Add accumulators to PySpark
2013-01-20 11:00:36 -08:00
Matei Zaharia
2a8c2a6790
Minor formatting fixes
2013-01-20 10:24:53 -08:00
Matei Zaharia
5d473f050e
Merge pull request #376 from MLnick/python-als
...
Python ALS example
2013-01-20 10:21:29 -08:00
Matei Zaharia
922c5ec069
Merge pull request #385 from pwendell/ec2-guide-fix
...
Clarifying log directory in EC2 guide
2013-01-20 10:05:38 -08:00
Patrick Wendell
5f74ead636
Changes based on Matei's comment
2013-01-20 08:59:20 -08:00
Tathagata Das
76ff962edc
Merge pull request #380 from tdas/streaming
...
Merging pySpark to streaming
2013-01-20 03:54:46 -08:00
Tathagata Das
33bad85bb9
Fixed streaming testsuite bugs
2013-01-20 03:51:11 -08:00
Matei Zaharia
ee5a07955c
Fix Python guide to say accumulators are available
2013-01-20 02:11:58 -08:00
Matei Zaharia
a23ed25f3c
Add a class comment to Accumulator
2013-01-20 02:10:25 -08:00
Matei Zaharia
61b6382a35
Launch accumulator tests in run-tests
2013-01-20 01:59:07 -08:00
Matei Zaharia
8e7f098a2c
Added accumulators to PySpark
2013-01-20 01:57:44 -08:00
Tathagata Das
4f8fe58b25
Merge branch 'mesos-streaming' into streaming
...
Conflicts:
core/src/main/scala/spark/api/java/JavaRDDLike.scala
core/src/main/scala/spark/api/java/JavaSparkContext.scala
core/src/test/scala/spark/JavaAPISuite.java
2013-01-20 01:13:56 -08:00
Tathagata Das
214345ceac
Fixed issue https://spark-project.atlassian.net/browse/STREAMING-29 , along with updates to doc comments in SparkContext.checkpoint().
2013-01-19 23:50:17 -08:00
Patrick Wendell
ecdff861f7
Clarifying log directory in EC2 guide
2013-01-19 22:59:35 -08:00
Imran Rashid
d98caa0fa0
Merge remote-tracking branch 'dennybritz/blockmanagerUI' into blockmanager_ui
...
Conflicts:
core/src/main/scala/spark/RDD.scala
core/src/main/scala/spark/storage/BlockManagerMaster.scala
core/src/main/scala/spark/storage/StorageLevel.scala
2013-01-18 18:11:26 -08:00
Patrick Wendell
11bbe23140
Merge pull request #369 from pwendell/streaming-java-api
...
Java API For Spark Streaming
2013-01-17 22:39:30 -08:00
Patrick Wendell
12b72b3e73
NetworkWordCount example
2013-01-17 22:37:56 -08:00
Patrick Wendell
c46dd2de78
Moving tests to appropriate directory
2013-01-17 21:43:17 -08:00
Patrick Wendell
e0165bf714
Adding queueStream and some slight refactoring
2013-01-17 21:25:49 -08:00
Patrick Wendell
6fba7683c2
Small doc fix
2013-01-17 18:46:24 -08:00
Patrick Wendell
ee0314c3b3
Merge branch 'streaming' into streaming-java-api
2013-01-17 18:43:00 -08:00
Patrick Wendell
70ba994d6d
Import fixup
2013-01-17 18:41:59 -08:00
Patrick Wendell
2261e62ee5
Style cleanup
2013-01-17 18:41:59 -08:00
Patrick Wendell
82b8707c6b
Checkpointing in Streaming java API
2013-01-17 18:41:58 -08:00
Patrick Wendell
61b877c688
Adding flatMap
2013-01-17 18:41:58 -08:00
Patrick Wendell
d5570c7968
Adding checkpointing to Java API
2013-01-17 18:41:58 -08:00
Patrick Wendell
8e6cbbc6c7
Adding other updateState functions
2013-01-17 18:41:58 -08:00
Patrick Wendell
2a872335c5
Bug fix and test cleanup
2013-01-17 18:41:58 -08:00
Matei Zaharia
54c0f9f185
Fix code that assumed spark.local.dir is only a single directory
2013-01-17 17:40:55 -08:00
Matei Zaharia
b534fd363f
Merge pull request #382 from fanuo/master
...
HttpBroadcast server cache by default in spark.local.dir instead of java.io.tmpdir
2013-01-17 17:00:25 -08:00
Fernand Pajot
742bc841ad
changed HttpBroadcast server cache to be in spark.local.dir instead of java.io.tmpdir
2013-01-17 16:56:11 -08:00
Matei Zaharia
46644e409d
Merge branch 'master' of github.com:mesos/spark
2013-01-17 11:17:19 -08:00
Matei Zaharia
892c32a14b
Warn users if they run pyspark or spark-shell without compiling Spark
2013-01-17 11:14:47 -08:00
Nick Pentreath
a5ba7a9f32
Use only one update function and pass in transpose of ratings matrix where appropriate
2013-01-17 16:21:00 +02:00
Nick Pentreath
a512df551f
Fixed index error missing first argument
2013-01-17 16:05:27 +02:00
Nick Pentreath
42fbef3c2a
Adding default command line args to SparkALS
2013-01-17 15:54:59 +02:00
Matei Zaharia
aff1844155
Merge pull request #381 from squito/remove_threadpool
...
remove unused thread pool
2013-01-16 16:46:42 -08:00
Tathagata Das
f466ee44bc
Merge branch 'master' into streaming
...
Conflicts:
core/src/main/scala/spark/MapOutputTracker.scala
2013-01-16 12:57:11 -08:00
Imran Rashid
eae698f755
remove unused thread pool
2013-01-16 12:21:37 -08:00
Tathagata Das
a805ac4a7c
Disabled checkpoint for PairwiseRDD (pySpark).
2013-01-16 10:55:26 -08:00
Matei Zaharia
4beb084f64
Merge pull request #374 from woggling/null-mapout
...
Generate FetchFailedException even for cached missing map outputs
2013-01-15 14:22:29 -08:00
Matei Zaharia
7adfedb0d7
Merge pull request #378 from apsaltis/master
...
Updated SCALA_VERSION in run2.cmd to match runtime version of Scala
2013-01-15 14:20:44 -08:00