Reynold Xin
4a31877408
Added the unpersist api to JavaRDD.
2013-05-01 20:31:54 -07:00
Reynold Xin
98df9d2853
Added removeRdd function in BlockManager.
2013-05-01 20:17:09 -07:00
Reynold Xin
204eb32e14
Changed the type of the persistentRdds hashmap back to
...
TimeStampedHashMap.
2013-05-01 16:14:58 -07:00
Reynold Xin
34637b97ec
Added SparkContext.cleanup back. Not sure why it was removed before ...
2013-05-01 16:12:37 -07:00
Reynold Xin
3227ec8edd
Cleaned up Ram's code. Moved SparkContext.remove to RDD.unpersist.
...
Also updated unit tests to make sure they are properly testing for
concurrency.
2013-05-01 16:07:44 -07:00
harshars
8481562731
Merged Ram's commit on removing RDDs.
...
Conflicts:
core/src/main/scala/spark/SparkContext.scala
2013-05-01 14:42:17 -07:00
Matei Zaharia
aa8fe1a209
Merge pull request #586 from mridulm/master
...
Pull request to address issues Reynold Xin reported
2013-04-30 22:30:18 -07:00
Mridul Muralidharan
60cabb35cb
Add addition catch block for exception too
2013-05-01 01:17:14 +05:30
Mridul Muralidharan
3b748ced22
Be more aggressive and defensive in all uses of SelectionKey in select loop
2013-05-01 00:30:30 +05:30
Mridul Muralidharan
0f45477be1
Change indentation
2013-05-01 00:10:02 +05:30
Mridul Muralidharan
538614acfe
Be more aggressive and defensive in select also
2013-05-01 00:05:32 +05:30
Mridul Muralidharan
48854e1dbf
If key is not valid, close connection
2013-04-30 23:59:33 +05:30
Matei Zaharia
f708dda81e
Merge pull request #585 from pwendell/listener-perf
...
[Fix SPARK-742] Task Metrics should not employ per-record timing by default
2013-04-30 07:51:40 -07:00
Mridul Muralidharan
e46d547ccd
Fix issues reported by Reynold
2013-04-30 16:15:56 +05:30
Patrick Wendell
016ce1fa9c
Using full package name for util
2013-04-29 12:02:27 -07:00
Patrick Wendell
540be6b154
Modified version of the fix which just removes all per-record tracking.
2013-04-29 11:32:07 -07:00
Patrick Wendell
224fbac061
Spark-742: TaskMetrics should not employ per-record timing.
...
This patch does three things:
1. Makes TimedIterator a trait with two implementations (one a no-op)
2. Makes the default behavior to use the no-op implementation
3. Removes DelegateBlockFetchTracker. This is just cleanup, but it seems like
the triat doesn't really reduce complexity in any way.
In the future we can add other implementations, e.g. ones which perform sampling.
2013-04-29 11:13:43 -07:00
Shivaram Venkataraman
604d3bf56c
Rename partition class and add scala doc
2013-04-28 16:31:07 -07:00
Shivaram Venkataraman
15acd49f07
Actually rename classes to ZippedPartitions*
...
(the previous commit only renamed the file)
2013-04-28 16:03:22 -07:00
Shivaram Venkataraman
6e84635ab9
Rename classes from MapZipped* to Zipped*
2013-04-28 15:58:40 -07:00
Shivaram Venkataraman
0cc6642b7c
Rename to zipPartitions and style changes
2013-04-28 05:11:03 -07:00
Shivaram Venkataraman
c9c4954d99
Add an interface to zip iterators of multiple RDDs
...
The current code supports 2, 3 or 4 arguments but can be extended
to more arguments if required.
2013-04-26 16:57:46 -07:00
Matei Zaharia
6e6b5204ea
Create an empty directory when checkpointing a 0-partition RDD (fixes a
...
test failure on Hadoop 2.0)
2013-04-25 00:42:37 -07:00
Mridul Muralidharan
dd515ca3ee
Attempt at fixing merge conflict
2013-04-24 09:24:17 +05:30
Mridul Muralidharan
7acab3ab45
Fix review comments, add a new api to SparkHadoopUtil to create appropriate Configuration. Modify an example to show how to use SplitInfo
2013-04-22 08:01:13 +05:30
Mridul Muralidharan
5ee2f5c483
Cache pattern, add (commented out) alternatives for check* apis
2013-04-17 23:13:34 +05:30
Mridul Muralidharan
f07961060d
Add a small note on spark.tasks.schedule.aggression
2013-04-17 23:13:02 +05:30
Mridul Muralidharan
02dffd2eb0
Ensure all ask/await block for spark.akka.askTimeout - so that it is controllable : instead of arbitrary timeouts spread across codebase. In our tests, we use 30 seconds, though default of 10 is maintained
2013-04-17 05:52:57 +05:30
Mridul Muralidharan
ad80f68eb5
remove spurious debug statements
2013-04-16 22:15:34 +05:30
Mridul Muralidharan
f7969f72ee
Fix exception when checkpoint path does not exist (no data in rdd which is being checkpointed for example)
2013-04-16 21:51:38 +05:30
Mridul Muralidharan
323ab8ff3b
Scala does not prevent variable shadowing ! Sick error due to it ...
2013-04-16 17:05:10 +05:30
Mridul Muralidharan
59c380d69a
Fix npe
2013-04-16 03:29:38 +05:30
Mridul Muralidharan
dd2b64ec97
Fix bug with atomic update
2013-04-16 03:19:24 +05:30
Mridul Muralidharan
5540ab8243
Use hostname instead of hostport for executor, fix creation of workdir
2013-04-16 02:57:43 +05:30
Mridul Muralidharan
eb7e95e833
Commit job to persist files
2013-04-16 02:56:36 +05:30
Matei Zaharia
a64c107449
Make ShuffledRDD.prev transient
2013-04-15 16:41:51 -04:00
Mridul Muralidharan
19652a44be
Fix issue with FileSuite failing
2013-04-15 19:16:36 +05:30
Mridul Muralidharan
54b3d45b81
Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues
2013-04-15 18:26:50 +05:30
Mridul Muralidharan
d90d2af103
Checkpoint commit - compiles and passes a lot of tests - not all though, looking into FileSuite issues
2013-04-15 18:12:11 +05:30
Matei Zaharia
c35d530bcf
Fix compile error
2013-04-13 12:43:12 -04:00
Andrew Ash
29d3440efb
Add details when BlockManager heartbeats time out
...
Makes it more clear what the threshold was for tuning spark.storage.blockManagerSlaveTimeoutMs
Before:
WARN "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats
After:
WARN "Removing BlockManager BlockManagerId(201304022120-1976232532-5050-27464-0, myhostname, 51337) with no recent heart beats: 19216ms exceeds 15000ms
2013-04-11 01:54:02 -03:00
Matei Zaharia
054feb6448
Fixed a bug with zip
2013-04-07 21:15:21 -04:00
Matei Zaharia
b5900d47b1
Fix compile warning
2013-04-07 20:55:42 -04:00
Matei Zaharia
6962d40b44
Fix deprecated warning
2013-04-07 20:27:33 -04:00
Mridul Muralidharan
6798a09df8
Add support for building against hadoop2-yarn : adding new maven profile for it
2013-04-07 17:47:38 +05:30
Mark Hamstra
e215f67923
Correct sense of 'filter out' in comment.
2013-03-31 08:00:13 -07:00
Mark Hamstra
8bcdc64005
Fixed broken filter in getWritableClass[T]
2013-03-30 22:09:52 -07:00
Matei Zaharia
9831bc1a09
Merge pull request #539 from cgrothaus/fix-webui-workdirpath
...
Bugfix: WorkerWebUI must respect workDirPath from Worker
2013-03-29 22:16:22 -07:00
Matei Zaharia
3cc8ab6e29
Merge pull request #541 from stephenh/shufflecoalesce
...
Add a shuffle parameter to coalesce.
2013-03-29 22:14:07 -07:00
Holden Karau
f5df729b12
Explicitly catch all throwables (warning in 2.10)
2013-03-24 16:15:32 -07:00
