ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2891 commits 6 branches 181 tags 397 MiB
Commit graph

2891 commits

This branch
This branch
All branches
Author SHA1 Message Date
Mridul Muralidharan 609a817f52 Integrate review comments on pull request 2013-05-02 06:44:33 +05:30
Jey Kottalam 207afe4088 Remove spark-repl's extraneous dependency on spark-streaming 2013-05-01 16:57:31 -07:00
Reynold Xin 204eb32e14 Changed the type of the persistentRdds hashmap back to 
TimeStampedHashMap.
 2013-05-01 16:14:58 -07:00
Reynold Xin 34637b97ec Added SparkContext.cleanup back. Not sure why it was removed before ... 2013-05-01 16:12:37 -07:00
Reynold Xin 3227ec8edd Cleaned up Ram's code. Moved SparkContext.remove to RDD.unpersist. 
Also updated unit tests to make sure they are properly testing for
concurrency.
 2013-05-01 16:07:44 -07:00
harshars 8481562731 Merged Ram's commit on removing RDDs. 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
 2013-05-01 14:42:17 -07:00
Reynold Xin 1deee67615 Merge branch 'master' of github.com:mesos/spark 2013-05-01 14:03:24 -07:00
Mridul Muralidharan 27764a00f4 Fix some npe introduced accidentally 2013-05-01 20:56:05 +05:30
Mridul Muralidharan d960e7e0f8 a) Add support for hyper local scheduling - specific to a host + port - before trying host local scheduling. 
b) Add some fixes to test code to ensure it passes (and fixes some other issues).

c) Fix bug in task scheduling which incorrectly used availableCores instead of all cores on the node.
 2013-05-01 20:24:00 +05:30
Matei Zaharia aa8fe1a209 Merge pull request #586 from mridulm/master 
Pull request to address issues Reynold Xin reported
 2013-04-30 22:30:18 -07:00
Reynold Xin dd7bef3147 Two minor fixes according to Ryan LeCompte's review. 2013-04-30 15:02:32 -07:00
Reynold Xin cea6174573 Merge branch 'master' of github.com:mesos/spark into blockmanager 
Conflicts:
	core/src/main/scala/spark/BlockStoreShuffleFetcher.scala
 2013-04-30 13:28:35 -07:00
Reynold Xin 13bde4f2e5 Merge branch 'master' of github.com:mridulm/spark 2013-04-30 12:54:01 -07:00
Mridul Muralidharan 60cabb35cb Add addition catch block for exception too 2013-05-01 01:17:14 +05:30
Mridul Muralidharan c446ac31d7 Spurious commit, reverting gitignore change 2013-05-01 00:32:30 +05:30
Mridul Muralidharan 3b748ced22 Be more aggressive and defensive in all uses of SelectionKey in select loop 2013-05-01 00:30:30 +05:30
Reynold Xin 7cf13709c3 Merge branch 'master' of github.com:mridulm/spark 2013-04-30 11:46:01 -07:00
Mridul Muralidharan 0f45477be1 Change indentation 2013-05-01 00:10:02 +05:30
Mridul Muralidharan 538614acfe Be more aggressive and defensive in select also 2013-05-01 00:05:32 +05:30
Mridul Muralidharan 48854e1dbf If key is not valid, close connection 2013-04-30 23:59:33 +05:30
Reynold Xin 084df85816 Merge branch 'master' of github.com:mridulm/spark 2013-04-30 10:22:37 -07:00
Matei Zaharia f708dda81e Merge pull request #585 from pwendell/listener-perf 
[Fix SPARK-742] Task Metrics should not employ per-record timing by default
 2013-04-30 07:51:40 -07:00
Mridul Muralidharan e46d547ccd Fix issues reported by Reynold 2013-04-30 16:15:56 +05:30
Reynold Xin 1055785a83 Allow specifying the shuffle write file buffer size. The default buffer 
size is 8KB in FastBufferedOutputStream, which is too small and would
cause a lot of disk seeks.
 2013-04-29 23:33:56 -07:00
Reynold Xin 7007201201 Added a shuffle block manager so it is easier in the future to 
consolidate shuffle output files.
 2013-04-29 23:07:03 -07:00
Reynold Xin ed4ddf4487 Merge branch 'master' of github.com:mesos/spark into blockmanager 2013-04-29 20:08:23 -07:00
Matei Zaharia f1f92c88eb Build against Hadoop 1 by default 2013-04-29 17:08:45 -07:00
Reynold Xin d3586ef438 Merge branch 'blockmanager' of github.com:rxin/spark into blockmanager 
Conflicts:
	core/src/main/scala/spark/storage/DiskStore.scala
 2013-04-29 15:44:18 -07:00
Patrick Wendell 016ce1fa9c Using full package name for util 2013-04-29 12:02:27 -07:00
Patrick Wendell 540be6b154 Modified version of the fix which just removes all per-record tracking. 2013-04-29 11:32:07 -07:00
Patrick Wendell 224fbac061 Spark-742: TaskMetrics should not employ per-record timing. 
This patch does three things:

1. Makes TimedIterator a trait with two implementations (one a no-op)
2. Makes the default behavior to use the no-op implementation
3. Removes DelegateBlockFetchTracker. This is just cleanup, but it seems like
   the triat doesn't really reduce complexity in any way.

In the future we can add other implementations, e.g. ones which perform sampling.
 2013-04-29 11:13:43 -07:00
Matei Zaharia 0f45347c7b More unit test fixes 2013-04-28 22:29:27 -07:00
Matei Zaharia bce4089f22 Fix BlockManagerSuite to deal with clearing spark.hostPort 2013-04-28 22:23:48 -07:00
Matei Zaharia 68c07ea198 Merge pull request #582 from shivaram/master 
Add zip partitions interface
 2013-04-28 20:19:33 -07:00
Shivaram Venkataraman 604d3bf56c Rename partition class and add scala doc 2013-04-28 16:31:07 -07:00
Shivaram Venkataraman 15acd49f07 Actually rename classes to ZippedPartitions* 
(the previous commit only renamed the file)
 2013-04-28 16:03:22 -07:00
Shivaram Venkataraman 6e84635ab9 Rename classes from MapZipped* to Zipped* 2013-04-28 15:58:40 -07:00
Matei Zaharia f6ee9a8728 Merge pull request #583 from mridulm/master 
Fix issues with streaming test cases after yarn branch merge
 2013-04-28 15:36:04 -07:00
Mridul Muralidharan 430c531464 Remove debug statements 2013-04-29 00:24:30 +05:30
Mridul Muralidharan 3a89a76b87 Make log message more descriptive to aid in debugging 2013-04-29 00:04:12 +05:30
Mridul Muralidharan 9bd439502e Remove spurious commit 2013-04-28 23:09:08 +05:30
Mridul Muralidharan 7fa6978a1e Allow CheckpointWriter pending tasks to finish 2013-04-28 23:08:10 +05:30
Mridul Muralidharan 00c7a37604 Merge branch 'master' of github.com:mridulm/spark 2013-04-28 22:44:34 +05:30
Mridul Muralidharan afee902443 Attempt to fix streaming test failures after yarn branch merge 2013-04-28 22:26:45 +05:30
Shivaram Venkataraman 0cc6642b7c Rename to zipPartitions and style changes 2013-04-28 05:11:03 -07:00
Shivaram Venkataraman c9c4954d99 Add an interface to zip iterators of multiple RDDs 
The current code supports 2, 3 or 4 arguments but can be extended
to more arguments if required.
 2013-04-26 16:57:46 -07:00
Matei Zaharia 1f20ef2567 Merge branch 'master' of github.com:mesos/spark 2013-04-25 20:03:13 -07:00
Matei Zaharia 1b169f190c Exclude old versions of Netty, which had a different Maven organization 2013-04-25 19:52:12 -07:00
Matei Zaharia cf54b824ff Merge pull request #580 from pwendell/quickstart 
SPARK-739 Have quickstart standlone job use README
 2013-04-25 11:45:58 -07:00
Patrick Wendell a72134a6ac SPARK-739 Have quickstart standlone job use README 2013-04-25 10:39:28 -07:00