Stephen Haberman
8dc06069fe
Rename RDD.tupleBy to keyBy.
2013-01-06 15:21:45 -06:00
Stephen Haberman
1fdb6946b5
Add RDD.tupleBy.
2013-01-05 13:07:59 -06:00
Matei Zaharia
55809fbc6d
Merge pull request #349 from woggling/cache-finally
...
Avoid stalls when computation of cached RDD throws exception
2013-01-01 08:21:33 -08:00
Matei Zaharia
c593f6329e
Merge pull request #348 from JoshRosen/spark-597
...
Raise exception when hashing Java arrays (SPARK-597)
2013-01-01 08:20:06 -08:00
Charles Reiss
58072a7340
Remove some dead comments
2013-01-01 08:07:44 -08:00
Charles Reiss
21636ee4fa
Test with exception while computing cached RDD.
2013-01-01 08:07:40 -08:00
Charles Reiss
feadaf72f4
Mark key as not loading in CacheTracker even when compute() fails
2013-01-01 07:57:20 -08:00
Josh Rosen
f803953998
Raise exception when hashing Java arrays (SPARK-597)
2012-12-31 20:20:11 -08:00
Matei Zaharia
3f74f729a1
Merge pull request #345 from JoshRosen/fix/add-file
...
Fix deletion of files in current working directory by clearFiles()
2012-12-29 15:01:33 -08:00
Josh Rosen
397e67103c
Change Utils.fetchFile() warning to SparkException.
2012-12-28 17:37:13 -08:00
Josh Rosen
d64fa72d2e
Add addFile() and addJar() to JavaSparkContext.
2012-12-28 17:00:57 -08:00
Josh Rosen
bd237d4a9d
Add synchronization to LocalScheduler.updateDependencies().
2012-12-28 17:00:57 -08:00
Josh Rosen
f1bf4f0385
Skip deletion of files in clearFiles().
...
This fixes an issue where Spark could delete
original files in the current working directory
that were added to the job using addFile().
There was also the potential for addFile() to
overwrite local files, which is addressed by
changing Utils.fetchFile() to log a warning
instead of overwriting a file with new contents.
This is a short-term fix; a better long-term
solution would be to remove the dependence on
storing files in the current working directory,
since we can't change the cwd from Java.
2012-12-28 17:00:57 -08:00
Matei Zaharia
84587a9bf3
Merge pull request #343 from markhamstra/spark-601
...
lookup() needn't fail when there is no partitioner
2012-12-24 15:28:05 -08:00
Mark Hamstra
903f3518df
fall back to filter-map-collect when calling lookup() on an RDD without a partitioner
2012-12-24 13:18:45 -08:00
Matei Zaharia
b575cbe069
Merge pull request #342 from markhamstra/spark-645
...
Allow distinct() to be called without parentheses
2012-12-24 08:04:50 -08:00
Mark Hamstra
61be8566e2
Allow distinct() to be called without parentheses when using the default number of splits.
2012-12-24 02:36:47 -08:00
Reynold Xin
a6bb41c6d3
Updated Kryo version for Maven pom file.
2012-12-21 16:25:50 -08:00
Reynold Xin
c68a076037
Updated Kryo documentation for Kryo version update.
2012-12-21 16:03:17 -08:00
Reynold Xin
60f7338092
Remove the call to close input stream in Kryo serializer.
2012-12-21 15:49:33 -08:00
Matei Zaharia
3334b7c6b5
Merge pull request #341 from rxin/4a3fb06ac2d11125feb08acbbd4df76d1e91b677
...
Kryo2 update against Spark master
2012-12-21 15:31:23 -08:00
Matei Zaharia
5e51b889fe
Merge pull request #327 from rxin/spark-633
...
Added the ability in block manager to remove blocks.
2012-12-20 11:33:38 -08:00
Reynold Xin
9397c5014e
Let the slave notify the master block removal.
2012-12-20 01:37:09 -08:00
Matei Zaharia
e7051767f7
Merge pull request #337 from pwendell/worker-liveness-ui
...
SPARK-616: Logging dead workers in Web UI.
2012-12-19 15:31:32 -08:00
Reynold Xin
68c52d80ec
Moved BlockManager's IdGenerator into BlockManager object. Removed some
...
excessive debug messages.
2012-12-19 15:27:23 -08:00
Matei Zaharia
30b47794da
Merge pull request #340 from tomdz/deb-packaging-tweaks
...
Tweaked debian packaging to be a bit more in line with debian standards
2012-12-19 12:07:03 -08:00
Thomas Dudziak
5488ac67c3
Tweaked debian packaging to be a bit more in line with debian standards
2012-12-19 10:20:43 -08:00
Matei Zaharia
1e6e154d6d
Merge pull request #338 from tomdz/repl-pom-fix
...
Fixed repl maven build
2012-12-18 14:03:29 -08:00
Thomas Dudziak
4af6cad37a
Fixed repl maven build to produce artifacts with the appropriate hadoop classifier and extracted repl fat-jar and debian packaging into a separate project to make Maven happy
2012-12-18 12:08:19 -08:00
Patrick Wendell
bfac06e1f6
SPARK-616: Logging dead workers in Web UI.
...
This patch keeps track of which workers have died and marks them
as such in the master web UI. It also handles workers which die and
re-register using different actor ID's.
2012-12-17 23:09:05 -08:00
Matei Zaharia
b82a6dd2c7
Merge pull request #332 from JoshRosen/spark-607
...
Add try-finally to handle MapOutputTracker timeouts
2012-12-14 11:41:16 -08:00
Reynold Xin
06f855c24d
Merge branch 'spark-633' of github.com:rxin/spark into spark-633
2012-12-14 00:27:24 -08:00
Reynold Xin
8c01295b85
Fixed conflicts from merging Charles' and TD's block manager changes.
2012-12-14 00:26:36 -08:00
Matei Zaharia
1072f970cc
Merge pull request #331 from woggling/deploy-exit-status
...
Have standalone cluster report exit codes to clients
2012-12-13 22:43:48 -08:00
Charles Reiss
c528932a41
Code review cleanup.
2012-12-13 22:37:16 -08:00
Charles Reiss
0aad42b5e7
Have standalone cluster report exit codes to clients. Addresses SPARK-639.
2012-12-13 22:37:16 -08:00
Reynold Xin
0235667f73
Merge branch 'master' of github.com:mesos/spark into spark-633
2012-12-13 22:33:41 -08:00
Reynold Xin
97434f49b8
Merged TD's block manager refactoring.
2012-12-13 22:32:19 -08:00
Matei Zaharia
d6d910471d
Merge pull request #333 from rxin/master
...
Fixed the broken Java unit test from SPARK-635.
2012-12-13 22:31:53 -08:00
Reynold Xin
f4a9e1b9be
Fixed the broken Java unit test from SPARK-635.
2012-12-13 22:22:12 -08:00
Reynold Xin
41e58a519a
Merge branch 'master' of github.com:mesos/spark into spark-633
2012-12-13 22:06:47 -08:00
Josh Rosen
cf52d9cade
Add try-finally to handle MapOutputTracker timeouts.
2012-12-13 21:53:30 -08:00
Matei Zaharia
05e225f988
Merge pull request #329 from woggling/executor-status-codes
...
Executor exit status codes
2012-12-13 20:14:10 -08:00
Charles Reiss
b054d3b222
ExecutorLostReason -> ExecutorLossReason
2012-12-13 18:44:07 -08:00
Charles Reiss
24d7aa2d15
Extra whitespace in ExecutorExitCode
2012-12-13 18:39:23 -08:00
Matei Zaharia
012aa4e3d4
Merge pull request #330 from JoshRosen/spark-638
...
Use spark-env.sh to configure standalone master
2012-12-13 18:05:32 -08:00
Josh Rosen
1948f46093
Use spark-env.sh to configure standalone master. See SPARK-638.
...
Also fixed a typo in the standalone mode documentation.
2012-12-14 01:20:00 +00:00
Reynold Xin
dc7d7fc286
Merge branch 'master' of github.com:mesos/spark into spark-633
2012-12-13 16:48:34 -08:00
Matei Zaharia
dfcbb2ad97
Merge pull request #328 from rxin/spark-635
...
SPARK-635: Pass a TaskContext object to compute() interface and use that to close Hadoop input stream.
2012-12-13 16:46:54 -08:00
Reynold Xin
4f076e105e
SPARK-635: Pass a TaskContext object to compute() interface and use
...
that to close Hadoop input stream. Incorporated Matei's command.
2012-12-13 16:41:15 -08:00