ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
1960 commits 6 branches 181 tags 397 MiB
Commit graph

905 commits

Author SHA1 Message Date
Reynold Xin 636e912f32 Created a PruneDependency to properly assign dependency for 
SplitsPruningRDD.
 2013-01-23 21:21:55 -08:00
Reynold Xin 45cd50d5fe Updated assert == to ===. 2013-01-23 16:06:58 -08:00
Reynold Xin c24b3819dd Added an extra assert for split size check. 2013-01-23 15:34:59 -08:00
Reynold Xin eb222b7206 Added pruntSplits method to RDD. 2013-01-23 15:29:02 -08:00
Josh Rosen 43e9ff9596 Add test for driver hanging on exit (SPARK-530). 2013-01-22 22:47:26 -08:00
Matei Zaharia ebaa8f6519 Merge remote-tracking branch 'stephenh/cleanup' 
Conflicts:
	core/src/main/scala/spark/scheduler/local/LocalScheduler.scala
 2013-01-22 21:05:45 -08:00
Matei Zaharia d2d273868b Merge pull request #397 from JoshRosen/refactoring/daemon-threads 
Refactor daemon thread creation
 2013-01-22 21:02:53 -08:00
Matei Zaharia 151c47eef5 Merge pull request #399 from NFLabs/master 
Fix for hanging spark.HttpFileServer on the kind of virtual network
 2013-01-22 15:49:24 -08:00
Stephen Haberman 27b3f3f0a9 Handle slaveLost before slaveIdToHost knows about it. 2013-01-22 15:30:42 -06:00
Stephen Haberman 588b24197a Use default arguments instead of constructor overloads. 2013-01-22 10:19:30 -06:00
Leemoonsoo 7e9ee2e833 Fix for hanging spark.HttpFileServer with kind of virtual network 2013-01-22 23:08:34 +09:00
Josh Rosen 551a47a620 Refactor daemon thread pool creation. 2013-01-21 23:31:00 -08:00
Stephen Haberman a8baeb9327 Further simplify getOrElse call. 2013-01-21 21:30:24 -06:00
Stephen Haberman 2d8218b871 Remove unneeded/now-broken saveAsNewAPIHadoopFile overload. 2013-01-21 20:00:27 -06:00
Stephen Haberman ffd1623595 Minor cleanup. 2013-01-21 15:55:46 -06:00
Matei Zaharia a88b44ed3b Only bind to IPv4 addresses when trying to auto-detect external IP 2013-01-21 11:59:21 -08:00
Matei Zaharia 4d34c7fc3e Fix compile error caused by cherry-pick 2013-01-21 11:33:48 -08:00
Imran Rashid a3f571b539 more File -> String changes 2013-01-21 11:21:52 -08:00
Imran Rashid fe26acc482 remove unused imports 2013-01-21 11:21:46 -08:00
Imran Rashid c73107500e send sparkHome as String instead of File over network 2013-01-21 11:21:39 -08:00
Imran Rashid 5bf73df7f0 oops, fix stupid compile error 2013-01-21 11:21:33 -08:00
Imran Rashid aae5a920a4 get sparkHome the correct way 2013-01-21 11:21:28 -08:00
Imran Rashid f116d6b5c6 executor can use a different sparkHome from Worker 2013-01-21 11:21:22 -08:00
Stephen Haberman 6ded481999 Merge branch 'master' into hadoopconf 
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/api/java/JavaSparkContext.scala
 2013-01-21 12:56:48 -06:00
Stephen Haberman 69a417858b Also use hadoopConfiguration in newAPI methods. 2013-01-21 12:42:11 -06:00
Matei Zaharia c0b9ceb8c3 Log remote lifecycle events in Akka for easier debugging 2013-01-21 00:23:53 -08:00
Matei Zaharia c7b5e5f1ec Merge pull request #389 from JoshRosen/python_rdd_checkpointing 
Add checkpointing to the Python API
 2013-01-20 17:10:44 -08:00
Josh Rosen 9f211dd3f0 Fix PythonPartitioner equality; see SPARK-654. 
PythonPartitioner did not take the Python-side partitioning function
into account when checking for equality, which might cause problems
in the future.
 2013-01-20 15:41:42 -08:00
Josh Rosen 5b6ea9e9a0 Update checkpointing API docs in Python/Java. 2013-01-20 15:31:41 -08:00
Josh Rosen 7ed1bf4b48 Add RDD checkpointing to Python API. 2013-01-20 13:19:19 -08:00
Matei Zaharia 86057ec7c8 Merge branch 'master' into streaming 
Conflicts:
	core/src/main/scala/spark/api/python/PythonRDD.scala
 2013-01-20 12:47:55 -08:00
Matei Zaharia 8e7f098a2c Added accumulators to PySpark 2013-01-20 01:57:44 -08:00
Tathagata Das 4f8fe58b25 Merge branch 'mesos-streaming' into streaming 
Conflicts:
	core/src/main/scala/spark/api/java/JavaRDDLike.scala
	core/src/main/scala/spark/api/java/JavaSparkContext.scala
	core/src/test/scala/spark/JavaAPISuite.java
 2013-01-20 01:13:56 -08:00
Tathagata Das 214345ceac Fixed issue https://spark-project.atlassian.net/browse/STREAMING-29, along with updates to doc comments in SparkContext.checkpoint(). 2013-01-19 23:50:17 -08:00
Patrick Wendell ee0314c3b3 Merge branch 'streaming' into streaming-java-api 2013-01-17 18:43:00 -08:00
Patrick Wendell d5570c7968 Adding checkpointing to Java API 2013-01-17 18:41:58 -08:00
Matei Zaharia 54c0f9f185 Fix code that assumed spark.local.dir is only a single directory 2013-01-17 17:40:55 -08:00
Fernand Pajot 742bc841ad changed HttpBroadcast server cache to be in spark.local.dir instead of java.io.tmpdir 2013-01-17 16:56:11 -08:00
Matei Zaharia aff1844155 Merge pull request #381 from squito/remove_threadpool 
remove unused thread pool
 2013-01-16 16:46:42 -08:00
Tathagata Das f466ee44bc Merge branch 'master' into streaming 
Conflicts:
	core/src/main/scala/spark/MapOutputTracker.scala
 2013-01-16 12:57:11 -08:00
Imran Rashid eae698f755 remove unused thread pool 2013-01-16 12:21:37 -08:00
Tathagata Das a805ac4a7c Disabled checkpoint for PairwiseRDD (pySpark). 2013-01-16 10:55:26 -08:00
Matei Zaharia 4beb084f64 Merge pull request #374 from woggling/null-mapout 
Generate FetchFailedException even for cached missing map outputs
 2013-01-15 14:22:29 -08:00
Tathagata Das cd1521cfdb Merge branch 'master' into streaming 
Conflicts:
	core/src/main/scala/spark/rdd/CoGroupedRDD.scala
	core/src/main/scala/spark/rdd/FilteredRDD.scala
	docs/_layouts/global.html
	docs/index.md
	run
 2013-01-15 12:08:51 -08:00
Charles Reiss 4078623b9f Remove broken attempt to test fetching case. 2013-01-15 12:05:54 -08:00
Stephen Haberman d228bff440 Add a test. 2013-01-15 11:48:50 -06:00
Stephen Haberman dd583b7ebf Call executeOnCompleteCallbacks in a finally block. 2013-01-15 10:52:06 -06:00
Tathagata Das eded21925a Merge pull request #375 from tdas/streaming 
Important bug fixes
 2013-01-14 23:06:40 -08:00
Charles Reiss b038999797 Fix accidental spark.master.host reuse 2013-01-14 17:04:44 -08:00
Charles Reiss 7ba34bc007 Additional tests for MapOutputTracker. 2013-01-14 15:27:02 -08:00