Matei Zaharia
538ee755b4
Merge pull request #581 from jerryshao/master
...
fix [SPARK-740] block manage UI throws exception when enabling Spark Streaming
2013-05-02 09:01:42 -07:00
Charles Reiss
c847dd3da2
Don't accept generated temp directory names that can't be created successfully.
2013-05-01 23:19:10 -07:00
Reynold Xin
d13784b81e
Merge branch 'master' of github.com:mesos/spark
2013-05-01 22:50:25 -07:00
Matei Zaharia
9abcbcc76d
Merge pull request #591 from rxin/removerdd
...
RDD.unpersist: probably the most desired feature of Spark
2013-05-01 22:45:10 -07:00
Reynold Xin
4a31877408
Added the unpersist api to JavaRDD.
2013-05-01 20:31:54 -07:00
Reynold Xin
98df9d2853
Added removeRdd function in BlockManager.
2013-05-01 20:17:09 -07:00
Mridul Muralidharan
dfde9ce9dd
comment out debug versions of checkHost, etc from Utils - which were used to test
2013-05-02 07:41:33 +05:30
Mridul Muralidharan
1b5aaeadc7
Integrate review comments 2
2013-05-02 07:30:06 +05:30
jerryshao
c047f0e3ad
filter out Spark streaming block RDD and sort RDDInfo with id
2013-05-02 09:48:32 +08:00
Mridul Muralidharan
609a817f52
Integrate review comments on pull request
2013-05-02 06:44:33 +05:30
Jey Kottalam
207afe4088
Remove spark-repl's extraneous dependency on spark-streaming
2013-05-01 16:57:31 -07:00
Reynold Xin
204eb32e14
Changed the type of the persistentRdds hashmap back to
...
TimeStampedHashMap.
2013-05-01 16:14:58 -07:00
Reynold Xin
34637b97ec
Added SparkContext.cleanup back. Not sure why it was removed before ...
2013-05-01 16:12:37 -07:00
Reynold Xin
3227ec8edd
Cleaned up Ram's code. Moved SparkContext.remove to RDD.unpersist.
...
Also updated unit tests to make sure they are properly testing for
concurrency.
2013-05-01 16:07:44 -07:00
harshars
8481562731
Merged Ram's commit on removing RDDs.
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
2013-05-01 14:42:17 -07:00
Reynold Xin
1deee67615
Merge branch 'master' of github.com:mesos/spark
2013-05-01 14:03:24 -07:00
Mridul Muralidharan
27764a00f4
Fix some npe introduced accidentally
2013-05-01 20:56:05 +05:30
Mridul Muralidharan
d960e7e0f8
a) Add support for hyper local scheduling - specific to a host + port - before trying host local scheduling.
...
b) Add some fixes to test code to ensure it passes (and fixes some other issues).
c) Fix bug in task scheduling which incorrectly used availableCores instead of all cores on the node.
2013-05-01 20:24:00 +05:30
Prashant Sharma
dbe2887da7
Fixed deprecated method warning
2013-05-01 13:22:49 +05:30
Prashant Sharma
4041a2689e
Updated to latest stable scala 2.10.1 and akka 2.1.2
2013-05-01 11:35:35 +05:30
Prashant Sharma
e8a9d1cdf9
Fixed Warning: expect -> expectResult
2013-05-01 11:35:02 +05:30
Matei Zaharia
aa8fe1a209
Merge pull request #586 from mridulm/master
...
Pull request to address issues Reynold Xin reported
2013-04-30 22:30:18 -07:00
unknown
1d54401d7e
Modified as per TD's suggestions
2013-04-30 23:01:32 -06:00
Reynold Xin
dd7bef3147
Two minor fixes according to Ryan LeCompte's review.
2013-04-30 15:02:32 -07:00
Reynold Xin
cea6174573
Merge branch 'master' of github.com:mesos/spark into blockmanager
...
Conflicts:
core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
2013-04-30 13:28:35 -07:00
Reynold Xin
13bde4f2e5
Merge branch 'master' of github.com:mridulm/spark
2013-04-30 12:54:01 -07:00
Mridul Muralidharan
60cabb35cb
Add addition catch block for exception too
2013-05-01 01:17:14 +05:30
Mridul Muralidharan
c446ac31d7
Spurious commit, reverting gitignore change
2013-05-01 00:32:30 +05:30
Mridul Muralidharan
3b748ced22
Be more aggressive and defensive in all uses of SelectionKey in select loop
2013-05-01 00:30:30 +05:30
Reynold Xin
7cf13709c3
Merge branch 'master' of github.com:mridulm/spark
2013-04-30 11:46:01 -07:00
Mridul Muralidharan
0f45477be1
Change indentation
2013-05-01 00:10:02 +05:30
Mridul Muralidharan
538614acfe
Be more aggressive and defensive in select also
2013-05-01 00:05:32 +05:30
Mridul Muralidharan
48854e1dbf
If key is not valid, close connection
2013-04-30 23:59:33 +05:30
Reynold Xin
084df85816
Merge branch 'master' of github.com:mridulm/spark
2013-04-30 10:22:37 -07:00
Matei Zaharia
f708dda81e
Merge pull request #585 from pwendell/listener-perf
...
[Fix SPARK-742] Task Metrics should not employ per-record timing by default
2013-04-30 07:51:40 -07:00
Mridul Muralidharan
e46d547ccd
Fix issues reported by Reynold
2013-04-30 16:15:56 +05:30
Reynold Xin
1055785a83
Allow specifying the shuffle write file buffer size. The default buffer
...
size is 8KB in FastBufferedOutputStream, which is too small and would
cause a lot of disk seeks.
2013-04-29 23:33:56 -07:00
Reynold Xin
7007201201
Added a shuffle block manager so it is easier in the future to
...
consolidate shuffle output files.
2013-04-29 23:07:03 -07:00
Reynold Xin
ed4ddf4487
Merge branch 'master' of github.com:mesos/spark into blockmanager
2013-04-29 20:08:23 -07:00
Matei Zaharia
f1f92c88eb
Build against Hadoop 1 by default
2013-04-29 17:08:45 -07:00
Reynold Xin
d3586ef438
Merge branch 'blockmanager' of github.com:rxin/spark into blockmanager
...
Conflicts:
core/src/main/scala/spark/storage/DiskStore.scala
2013-04-29 15:44:18 -07:00
Patrick Wendell
016ce1fa9c
Using full package name for util
2013-04-29 12:02:27 -07:00
Patrick Wendell
540be6b154
Modified version of the fix which just removes all per-record tracking.
2013-04-29 11:32:07 -07:00
Patrick Wendell
224fbac061
Spark-742: TaskMetrics should not employ per-record timing.
...
This patch does three things:
1. Makes TimedIterator a trait with two implementations (one a no-op)
2. Makes the default behavior to use the no-op implementation
3. Removes DelegateBlockFetchTracker. This is just cleanup, but it seems like
the triat doesn't really reduce complexity in any way.
In the future we can add other implementations, e.g. ones which perform sampling.
2013-04-29 11:13:43 -07:00
Prashant Sharma
24bbf318b3
Fixied other warnings
2013-04-29 19:56:28 +05:30
Prashant Sharma
d3518f57cd
Fixed warning: erasure -> runtimeClass
2013-04-29 18:14:25 +05:30
Prashant Sharma
8f3ac240cb
Fixed Warning: ClassManifest -> ClassTag
2013-04-29 16:39:13 +05:30
Prashant Sharma
4b4a36ea7d
Fixed pom.xml with updated dependencies.
2013-04-29 12:55:43 +05:30
Matei Zaharia
0f45347c7b
More unit test fixes
2013-04-28 22:29:27 -07:00
Matei Zaharia
bce4089f22
Fix BlockManagerSuite to deal with clearing spark.hostPort
2013-04-28 22:23:48 -07:00