ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1987 commits 6 branches 181 tags 397 MiB
Commit graph

855 commits

Author SHA1 Message Date
Matei Zaharia fe5e4812fc Merge pull request #409 from rxin/splitpruningrdd 
Added pruntSplits method to RDD.
 2013-01-23 22:23:22 -08:00
Reynold Xin c109f29c97 Updated PruneDependency to change "split" to "partition". 2013-01-23 22:22:03 -08:00
Reynold Xin eedc542a02 Removed pruneSplits method in RDD and renamed SplitsPruningRDD to 
PartitionPruningRDD.
 2013-01-23 22:14:23 -08:00
Reynold Xin 81004b967e Marked prev RDD as transient in SplitsPruningRDD. 2013-01-23 21:54:27 -08:00
Reynold Xin 636e912f32 Created a PruneDependency to properly assign dependency for 
SplitsPruningRDD.
 2013-01-23 21:21:55 -08:00
Matei Zaharia 548856a224 Merge remote-tracking branch 'woggling/remove-machines' 
Conflicts:
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
 2013-01-23 15:44:17 -08:00
Reynold Xin eb222b7206 Added pruntSplits method to RDD. 2013-01-23 15:29:02 -08:00
Matei Zaharia 1dd82743e0 Fix compile error due to cherry-pick 2013-01-23 13:07:27 -08:00
Imran Rashid e1985bfa04 be sure to set class loader of kryo instances 2013-01-23 12:51:09 -08:00
Charles Reiss be4a115a7e Clarify TODO. 2013-01-23 12:48:45 -08:00
Matei Zaharia 1a3aeeca23 Merge pull request #407 from woggling/no-cache-tracker 
Eliminate CacheTracker
 2013-01-23 12:28:48 -08:00
Charles Reiss e1027ca639 Actually add CacheManager. 2013-01-23 12:22:11 -08:00
Matei Zaharia 4147e1d47b Merge pull request #406 from tdas/master 
Changed StorageLevel and BlockManagerId API to prevent duplication in memory
 2013-01-23 12:18:31 -08:00
Matei Zaharia 4d77d554e1 Merge pull request #394 from JoshRosen/add_file_fix 
Add SparkFiles.get() API to access files added through addFile().
 2013-01-23 12:16:30 -08:00
Josh Rosen ae2ed2947d Allow PySpark's SparkFiles to be used from driver 
Fix minor documentation formatting issues.
 2013-01-23 10:58:50 -08:00
Tathagata Das 79d55700ce One more fix. Made even default constructor of BlockManagerId private to prevent such problems in the future. 2013-01-23 01:57:09 -08:00
Charles Reiss d209b6b764 Extra debugging from hostLost() 2013-01-23 01:35:14 -08:00
Charles Reiss 9a27062260 Force generation increment after shuffle map stage 2013-01-23 01:34:44 -08:00
Tathagata Das 155f31398d Made StorageLevel constructor private, and added StorageLevels.create() to the Java API. Updates scala and java programming guides. 2013-01-23 01:10:26 -08:00
Tathagata Das 5e11f1e51f Modified StorageLevel API to ensure zero duplicate objects. 2013-01-22 23:42:53 -08:00
Tathagata Das bacade6caf Modified BlockManagerId API to ensure zero duplicate objects. Fixed BlockManagerId testcase in BlockManagerTestSuite. 2013-01-22 22:55:26 -08:00
Charles Reiss 2849931000 Eliminate CacheTracker. 
Replaces DAGScheduler's queries of CacheTracker with BlockManagerMaster
queries.

Adds CacheManager to locally coordinate computation of cached RDDs.
 2013-01-22 22:19:30 -08:00
Matei Zaharia ebaa8f6519 Merge remote-tracking branch 'stephenh/cleanup' 
Conflicts:
	core/src/main/scala/spark/scheduler/local/LocalScheduler.scala
 2013-01-22 21:05:45 -08:00
Matei Zaharia d2d273868b Merge pull request #397 from JoshRosen/refactoring/daemon-threads 
Refactor daemon thread creation
 2013-01-22 21:02:53 -08:00
Matei Zaharia 151c47eef5 Merge pull request #399 from NFLabs/master 
Fix for hanging spark.HttpFileServer on the kind of virtual network
 2013-01-22 15:49:24 -08:00
Stephen Haberman 27b3f3f0a9 Handle slaveLost before slaveIdToHost knows about it. 2013-01-22 15:30:42 -06:00
Stephen Haberman 588b24197a Use default arguments instead of constructor overloads. 2013-01-22 10:19:30 -06:00
Leemoonsoo 7e9ee2e833 Fix for hanging spark.HttpFileServer with kind of virtual network 2013-01-22 23:08:34 +09:00
Charles Reiss e353886a8c Use generation numbers for fetch failure tracking 2013-01-22 00:23:31 -08:00
Josh Rosen 551a47a620 Refactor daemon thread pool creation. 2013-01-21 23:31:00 -08:00
Stephen Haberman a8baeb9327 Further simplify getOrElse call. 2013-01-21 21:30:24 -06:00
Stephen Haberman 2d8218b871 Remove unneeded/now-broken saveAsNewAPIHadoopFile overload. 2013-01-21 20:00:27 -06:00
Josh Rosen 7b9e96c992 Add synchronization to Executor.updateDependencies() (SPARK-662) 2013-01-21 17:34:23 -08:00
Josh Rosen ef711902c1 Don't download files to master's working directory. 
This should avoid exceptions caused by existing
files with different contents.

I also removed some unused code.
 2013-01-21 17:34:17 -08:00
Stephen Haberman ffd1623595 Minor cleanup. 2013-01-21 15:55:46 -06:00
Matei Zaharia a88b44ed3b Only bind to IPv4 addresses when trying to auto-detect external IP 2013-01-21 11:59:21 -08:00
Matei Zaharia 4d34c7fc3e Fix compile error caused by cherry-pick 2013-01-21 11:33:48 -08:00
Imran Rashid a3f571b539 more File -> String changes 2013-01-21 11:21:52 -08:00
Imran Rashid fe26acc482 remove unused imports 2013-01-21 11:21:46 -08:00
Imran Rashid c73107500e send sparkHome as String instead of File over network 2013-01-21 11:21:39 -08:00
Imran Rashid 5bf73df7f0 oops, fix stupid compile error 2013-01-21 11:21:33 -08:00
Imran Rashid aae5a920a4 get sparkHome the correct way 2013-01-21 11:21:28 -08:00
Imran Rashid f116d6b5c6 executor can use a different sparkHome from Worker 2013-01-21 11:21:22 -08:00
Stephen Haberman 6ded481999 Merge branch 'master' into hadoopconf 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/api/java/JavaSparkContext.scala
 2013-01-21 12:56:48 -06:00
Stephen Haberman 69a417858b Also use hadoopConfiguration in newAPI methods. 2013-01-21 12:42:11 -06:00
Matei Zaharia c0b9ceb8c3 Log remote lifecycle events in Akka for easier debugging 2013-01-21 00:23:53 -08:00
Matei Zaharia c7b5e5f1ec Merge pull request #389 from JoshRosen/python_rdd_checkpointing 
Add checkpointing to the Python API
 2013-01-20 17:10:44 -08:00
Josh Rosen 9f211dd3f0 Fix PythonPartitioner equality; see SPARK-654. 
PythonPartitioner did not take the Python-side partitioning function
into account when checking for equality, which might cause problems
in the future.
 2013-01-20 15:41:42 -08:00
Josh Rosen 5b6ea9e9a0 Update checkpointing API docs in Python/Java. 2013-01-20 15:31:41 -08:00
Josh Rosen 7ed1bf4b48 Add RDD checkpointing to Python API. 2013-01-20 13:19:19 -08:00