ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2244 commits 6 branches 181 tags 397 MiB
Commit graph

2244 commits

This branch
This branch
All branches
Author SHA1 Message Date
Shivaram Venkataraman d012cfa0e8 Update spark_ec2.py to use new spark-ec2 scripts 2013-01-26 22:48:39 -08:00
Matei Zaharia 49f6472c0f Merge pull request #418 from woggling/reregister-deadlock 
Fix BlockManager reregistration deadlock; do BlockManager reregistration more asynchronously
 2013-01-26 18:59:02 -08:00
Charles Reiss 58fc6b2bed Handle duplicate registrations better. 2013-01-26 18:30:44 -08:00
Charles Reiss ad4232b4da Fix deadlock in BlockManager reregistration triggered by failed updates. 2013-01-26 18:30:38 -08:00
Matei Zaharia ec2dadb521 Merge pull request #417 from JoshRosen/spark-668 
Fix JavaRDDLike.flatMap(PairFlatMapFunction) (SPARK-668).
 2013-01-26 16:20:57 -08:00
Josh Rosen d49cf0e587 Fix JavaRDDLike.flatMap(PairFlatMapFunction) (SPARK-668). 
This workaround is easier than rewriting JavaRDDLike in Java.
 2013-01-26 16:13:18 -08:00
Imran Rashid 49c05608f5 add metadatacleaner for persisentRdd map 2013-01-25 17:04:16 -08:00
Matei Zaharia 2435b7b5b7 Merge pull request #416 from stephenh/morefinally 
Call executeOnCompleteCallbacks in more finally blocks.
 2013-01-25 15:33:26 -08:00
Stephen Haberman 8efbda0b17 Call executeOnCompleteCallbacks in more finally blocks. 2013-01-25 14:55:33 -06:00
Imran Rashid a1d9d1767d fixup 1cadaa1, changed api of map 2013-01-25 10:05:26 -08:00
Imran Rashid 1cadaa164e switch to TimeStampedHashMap for storing persistent Rdds 2013-01-25 09:30:21 -08:00
Imran Rashid 539491bbc3 code reformatting 2013-01-25 09:29:59 -08:00
Stephen Haberman 7dfb82a992 Replace old 'master' term with 'driver'. 2013-01-25 11:03:00 -06:00
Prashant Sharma 4496bf197b Improved document comment in example 2013-01-25 14:34:38 +05:30
Matei Zaharia 04bfee2d08 Merge pull request #411 from stephenh/localsparkcontext 
Add LocalSparkContext to manage common sc variable.
 2013-01-24 23:00:50 -08:00
Stephen Haberman ec43a51b38 Merge branch 'master' into localsparkcontext 
Conflicts:
	core/src/test/scala/spark/FileServerSuite.scala
	core/src/test/scala/spark/RDDSuite.scala
 2013-01-24 21:17:30 -06:00
Matei Zaharia 45e6dd65b2 Merge pull request #412 from pwendell/master-url-warning 
SPARK-541: Adding a warning for invalid Master URL
 2013-01-24 15:58:43 -08:00
Patrick Wendell b6fc6e6752 SPARK-541: Adding a warning for invalid Master URL 
Right now Spark silently parses master URL's which do not match any
known regex as a Mesos URL. The Mesos error message when an invalid URL gets
passed is really confusing, so this warns the user when the implicit
conversion is happening.
 2013-01-24 14:31:23 -08:00
Matei Zaharia a2f4891d1d Merge pull request #396 from JoshRosen/spark-653 
Make PySpark AccumulatorParam an abstract base class
 2013-01-24 13:05:03 -08:00
Stephen Haberman 230bda2047 Add LocalSparkContext to manage common sc variable. 2013-01-24 11:01:01 -06:00
Matei Zaharia 0fe173a3a5 Merge pull request #410 from rxin/splitpruningrdd 
Added a clearDependencies method in PartitionPruningRDD.
 2013-01-23 23:10:15 -08:00
Reynold Xin 67a43bc7e6 Added a clearDependencies method in PartitionPruningRDD. 2013-01-23 23:06:52 -08:00
Matei Zaharia fe5e4812fc Merge pull request #409 from rxin/splitpruningrdd 
Added pruntSplits method to RDD.
 2013-01-23 22:23:22 -08:00
Reynold Xin c109f29c97 Updated PruneDependency to change "split" to "partition". 2013-01-23 22:22:03 -08:00
Reynold Xin eedc542a02 Removed pruneSplits method in RDD and renamed SplitsPruningRDD to 
PartitionPruningRDD.
 2013-01-23 22:14:23 -08:00
Reynold Xin 81004b967e Marked prev RDD as transient in SplitsPruningRDD. 2013-01-23 21:54:27 -08:00
Reynold Xin 636e912f32 Created a PruneDependency to properly assign dependency for 
SplitsPruningRDD.
 2013-01-23 21:21:55 -08:00
Reynold Xin 45cd50d5fe Updated assert == to ===. 2013-01-23 16:06:58 -08:00
Matei Zaharia 548856a224 Merge remote-tracking branch 'woggling/remove-machines' 
Conflicts:
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
 2013-01-23 15:44:17 -08:00
Reynold Xin c24b3819dd Added an extra assert for split size check. 2013-01-23 15:34:59 -08:00
Reynold Xin eb222b7206 Added pruntSplits method to RDD. 2013-01-23 15:29:02 -08:00
Matei Zaharia 1dd82743e0 Fix compile error due to cherry-pick 2013-01-23 13:07:27 -08:00
Charles Reiss 5c7422292e Remove more dead code from test. 2013-01-23 12:59:51 -08:00
Imran Rashid e1985bfa04 be sure to set class loader of kryo instances 2013-01-23 12:51:09 -08:00
Charles Reiss be4a115a7e Clarify TODO. 2013-01-23 12:48:45 -08:00
Charles Reiss 88b9d240fd Remove dead code in test. 2013-01-23 12:40:38 -08:00
Matei Zaharia 1a3aeeca23 Merge pull request #407 from woggling/no-cache-tracker 
Eliminate CacheTracker
 2013-01-23 12:28:48 -08:00
Charles Reiss e1027ca639 Actually add CacheManager. 2013-01-23 12:22:11 -08:00
Matei Zaharia 4147e1d47b Merge pull request #406 from tdas/master 
Changed StorageLevel and BlockManagerId API to prevent duplication in memory
 2013-01-23 12:18:31 -08:00
Matei Zaharia 4d77d554e1 Merge pull request #394 from JoshRosen/add_file_fix 
Add SparkFiles.get() API to access files added through addFile().
 2013-01-23 12:16:30 -08:00
Josh Rosen b47d054cfc Remove use of abc.ABCMeta due to cloudpickle issue. 
cloudpickle runs into issues while pickling subclasses of AccumulatorParam,
which may be related to this Python issue:

    http://bugs.python.org/issue7689

This seems hard to fix and the ABCMeta wasn't necessary, so I removed it.
 2013-01-23 11:47:27 -08:00
Josh Rosen ae2ed2947d Allow PySpark's SparkFiles to be used from driver 
Fix minor documentation formatting issues.
 2013-01-23 10:58:50 -08:00
Tathagata Das 9c8ff1e55f Fixed checkpoint testcases 2013-01-23 07:31:49 -08:00
Tathagata Das 666ce431aa Added support for rescheduling unprocessed batches on master failure. 2013-01-23 03:15:36 -08:00
Tathagata Das 79d55700ce One more fix. Made even default constructor of BlockManagerId private to prevent such problems in the future. 2013-01-23 01:57:09 -08:00
Charles Reiss 0b506dd2ec Add tests of various node failure scenarios. 2013-01-23 01:38:15 -08:00
Charles Reiss d209b6b764 Extra debugging from hostLost() 2013-01-23 01:35:14 -08:00
Charles Reiss 9a27062260 Force generation increment after shuffle map stage 2013-01-23 01:34:44 -08:00
Tathagata Das 155f31398d Made StorageLevel constructor private, and added StorageLevels.create() to the Java API. Updates scala and java programming guides. 2013-01-23 01:10:26 -08:00
Tathagata Das 5e11f1e51f Modified StorageLevel API to ensure zero duplicate objects. 2013-01-22 23:42:53 -08:00