ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
3644 commits 6 branches 181 tags 397 MiB
Commit graph

150 commits

Author SHA1 Message Date
Matei Zaharia 6b043a6f11 Merge pull request #724 from dlyubimov/SPARK-826 
SPARK-826: fold(), reduce(), collect() always attempt to use java serialization
 2013-08-06 22:31:02 -07:00
Alexander Pivovarov a30866438b SHARK-94 Log the files computed by HadoopRDD and NewHadoopRDD 2013-08-05 21:48:43 -07:00
Dmitriy Lyubimov cb6be5bd7e Merge remote-tracking branch 'mesos/master' into SPARK-826 
Conflicts:
	core/src/main/scala/spark/scheduler/cluster/ClusterTaskSetManager.scala
	core/src/main/scala/spark/scheduler/local/LocalTaskSetManager.scala
	core/src/test/scala/spark/KryoSerializerSuite.scala
 2013-07-31 22:09:22 -07:00
Dmitriy Lyubimov 28f1550f01 More elegant rewrite of the same. 2013-07-31 21:41:00 -07:00
Dmitriy Lyubimov 7c52ecc6a4 (1) added reduce test case. 
(2) added nested streaming in ParallelCollectionRDD
(3) added kryo with fold test which still doesn't work
 2013-07-31 19:27:30 -07:00
Dmitriy Lyubimov 96664431cb IDEA flipped JavaSerialized import at some point to a wrong class. 2013-07-30 23:10:09 -07:00
Dmitriy Lyubimov c219fc94fd Minor, style 2013-07-30 22:08:39 -07:00
Dmitriy Lyubimov f4b4b8836e reverting back to one-by-one serialization for parallelize() 2013-07-30 19:00:58 -07:00
Dmitriy Lyubimov abada94ebf removing default constructor (not Externalizable any more) 2013-07-30 18:04:02 -07:00
Dmitriy Lyubimov 943c6590c9 realiging "extends" back manually 2013-07-30 18:01:35 -07:00
Dmitriy Lyubimov ca33b12e98 resetting wrap and continuation indent = 4 2013-07-30 17:51:44 -07:00
Dmitriy Lyubimov 984b56155a changing approaches for parallelize(): java serialization needs to avoid writing headers! 2013-07-30 17:36:59 -07:00
Dmitriy Lyubimov ef9529a943 refactoring using writeByteBuffer() from Utils. 2013-07-30 16:24:23 -07:00
Dmitriy Lyubimov 1bca91633e + bug fixes; 
test added

Conflicts:

	core/src/test/scala/spark/KryoSerializerSuite.scala
 2013-07-30 11:04:11 -07:00
Dmitriy Lyubimov 8e5cd041bb initial externalization of ParallelCollectionRDD's split 2013-07-29 19:02:53 -07:00
Reynold Xin cb366774c8 Merge pull request #738 from harsha2010/pruning 
Fix bug in Partition Pruning.
 2013-07-26 16:59:30 -07:00
harshars 392d7474fd Code review 2013-07-26 15:23:15 -07:00
harshars 72cf7ec0e5 Indentation 2013-07-26 15:16:41 -07:00
harshars 822aac8f5a Indentation 2013-07-26 15:10:32 -07:00
harshars 743fc4e7aa Fix Bug in Partition Pruning, index of Pruned Partitions should inherit from parent 2013-07-26 14:35:17 -07:00
Matei Zaharia af3c9d5042 Add Apache license headers and LICENSE and NOTICE files 2013-07-16 17:21:33 -07:00
Matei Zaharia 94871e4703 Merge pull request #655 from tgravescs/master 
Add support for running Spark on Yarn on a secure Hadoop Cluster
 2013-07-06 15:26:19 -07:00
Y.CORP.YAHOO.COM\tgraves 923cf92900 Rework from pull request. Removed --user option from Spark on Yarn Client, made the user of JAVA_HOME environment 
variable conditional on if its set, and created addCredentials in each of the SparkHadoopUtil classes
to only add the credentials when the profile is hadoop2-yarn.
 2013-07-02 21:18:59 -05:00
Matei Zaharia 71030ba3eb Merge pull request #654 from lyogavin/enhance_pipe 
fix typo and coding style in #638
 2013-06-19 15:21:03 -07:00
Gavin Li 0a2a9bce1e fix typo and coding style 2013-06-18 21:30:13 +00:00
jerryshao 1e9269c3ee reduce ZippedPartitionsRDD's getPreferredLocations complexity 2013-06-18 09:49:06 +08:00
Gavin Li e6ae049283 Merge remote-tracking branch 'upstream1/master' into enhance_pipe 2013-06-16 22:53:39 +00:00
Gavin Li fb6d733fa8 update according to comments 2013-06-16 22:32:55 +00:00
Reynold Xin 2cc188fd54 SPARK-774: cogroup should also disable map side combine by default 2013-06-14 00:10:54 -07:00
Gavin Li e179ff8a32 update according to comments 2013-06-05 22:41:05 +00:00
Gavin Li 9f84315c05 enhance pipe to support what we can do in hadoop streaming 2013-06-01 00:26:10 +00:00
jerryshao bd3ea8f2a6 fix CheckpointRDD getPreferredLocations java.io.FileNotFoundException 2013-05-24 14:26:19 +08:00
Reynold Xin f9d40a5848 Added a comment in JdbcRDD for example usage. 2013-05-14 23:29:57 -07:00
Reynold Xin 81ad2fa331 Merge branch 'jdbc' of github.com:koeninger/spark 
Conflicts:
	project/SparkBuild.scala
 2013-05-14 23:12:00 -07:00
Matei Zaharia 016ac86830 Merge pull request #601 from rxin/emptyrdd-master 
EmptyRDD (master branch 0.8)
 2013-05-13 21:45:36 -07:00
Cody Koeninger 3da2305ed0 code cleanup per rxin comments 2013-05-11 23:59:07 -05:00
Reynold Xin 0fd84965f6 Added EmptyRDD. 2013-05-06 15:40:34 -07:00
Mridul Muralidharan 25198d7e9e Merge branch 'master' of github.com:mridulm/spark 2013-05-04 20:45:56 +05:30
Mridul Muralidharan 5b011d18d7 Merge from master 2013-05-04 20:41:27 +05:30
Mridul Muralidharan edb57c8331 Add support for instance local in getPreferredLocations of ZippedPartitionsBaseRDD. Add comments to both ZippedPartitionsBaseRDD and ZippedRDD to better describe the potential problem with the approach 2013-05-04 19:47:45 +05:30
Mridul Muralidharan ea2a6f91d3 pull from master 2013-05-04 00:35:59 +05:30
Reynold Xin 2bc895a829 Updated according to Matei's code review comment. 2013-05-03 01:02:16 -07:00
Mridul Muralidharan 11589c39d9 Fix ZippedRDD as part Matei's suggestion 2013-05-03 12:23:30 +05:30
Mridul Muralidharan d960e7e0f8 a) Add support for hyper local scheduling - specific to a host + port - before trying host local scheduling. 
b) Add some fixes to test code to ensure it passes (and fixes some other issues).

c) Fix bug in task scheduling which incorrectly used availableCores instead of all cores on the node.
 2013-05-01 20:24:00 +05:30
Reynold Xin d3586ef438 Merge branch 'blockmanager' of github.com:rxin/spark into blockmanager 
Conflicts:
	core/src/main/scala/spark/storage/DiskStore.scala
 2013-04-29 15:44:18 -07:00
Shivaram Venkataraman 604d3bf56c Rename partition class and add scala doc 2013-04-28 16:31:07 -07:00
Shivaram Venkataraman 15acd49f07 Actually rename classes to ZippedPartitions* 
(the previous commit only renamed the file)
 2013-04-28 16:03:22 -07:00
Shivaram Venkataraman 6e84635ab9 Rename classes from MapZipped* to Zipped* 2013-04-28 15:58:40 -07:00
Shivaram Venkataraman 0cc6642b7c Rename to zipPartitions and style changes 2013-04-28 05:11:03 -07:00
Shivaram Venkataraman c9c4954d99 Add an interface to zip iterators of multiple RDDs 
The current code supports 2, 3 or 4 arguments but can be extended
to more arguments if required.
 2013-04-26 16:57:46 -07:00