ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
2135 commits 6 branches 181 tags 397 MiB
Commit graph

2135 commits

This branch
This branch
All branches
Author SHA1 Message Date
Charles Reiss 0b506dd2ec Add tests of various node failure scenarios. 2013-01-23 01:38:15 -08:00
Charles Reiss d209b6b764 Extra debugging from hostLost() 2013-01-23 01:35:14 -08:00
Charles Reiss 9a27062260 Force generation increment after shuffle map stage 2013-01-23 01:34:44 -08:00
Tathagata Das 155f31398d Made StorageLevel constructor private, and added StorageLevels.create() to the Java API. Updates scala and java programming guides. 2013-01-23 01:10:26 -08:00
Tathagata Das 5e11f1e51f Modified StorageLevel API to ensure zero duplicate objects. 2013-01-22 23:42:53 -08:00
Matei Zaharia 7c3a1bddb7 Merge pull request #405 from JoshRosen/spark-530 
Add test for driver hanging on exit (SPARK-530)
 2013-01-22 23:28:47 -08:00
Tathagata Das bacade6caf Modified BlockManagerId API to ensure zero duplicate objects. Fixed BlockManagerId testcase in BlockManagerTestSuite. 2013-01-22 22:55:26 -08:00
Josh Rosen 43e9ff9596 Add test for driver hanging on exit (SPARK-530). 2013-01-22 22:47:26 -08:00
Charles Reiss 2849931000 Eliminate CacheTracker. 
Replaces DAGScheduler's queries of CacheTracker with BlockManagerMaster
queries.

Adds CacheManager to locally coordinate computation of cached RDDs.
 2013-01-22 22:19:30 -08:00
Matei Zaharia ebaa8f6519 Merge remote-tracking branch 'stephenh/cleanup' 
Conflicts:
	core/src/main/scala/spark/scheduler/local/LocalScheduler.scala
 2013-01-22 21:05:45 -08:00
Matei Zaharia d2d273868b Merge pull request #397 from JoshRosen/refactoring/daemon-threads 
Refactor daemon thread creation
 2013-01-22 21:02:53 -08:00
Matei Zaharia 05f13280de Merge pull request #404 from mbautin/master-fix-avro-dependency-in-repl 
Add an Avro dependency to REPL to make it compile with Hadoop 2
 2013-01-22 19:59:52 -08:00
Mikhail Bautin 325297e5c3 Add an Avro dependency to REPL to make it compile with Hadoop 2 2013-01-22 18:11:51 -08:00
Josh Rosen 35168d9c89 Fix sys.path bug in PySpark SparkContext.addPyFile 2013-01-22 17:54:11 -08:00
Stephen Haberman 98d0b7747d Fix Worker logInfo about unknown executor. 2013-01-22 18:11:51 -06:00
Stephen Haberman 8c51322cd0 Don't bother creating an exception. 2013-01-22 18:09:10 -06:00
Stephen Haberman fdec42385a Fix SPARK_MEM in ExecutorRunner. 2013-01-22 18:01:12 -06:00
Stephen Haberman 2437f6741b Restore SPARK_MEM in executorEnvs. 2013-01-22 18:01:03 -06:00
Matei Zaharia 151c47eef5 Merge pull request #399 from NFLabs/master 
Fix for hanging spark.HttpFileServer on the kind of virtual network
 2013-01-22 15:49:24 -08:00
Matei Zaharia 49c78446c8 Merge pull request #402 from stephenh/handleslavelosttoosoon 
Handle slaveLost before slaveIdToHost knows about it.
 2013-01-22 15:48:37 -08:00
Stephen Haberman 250fe89679 Handle Master telling the Worker to kill an already-dead executor. 2013-01-22 16:29:05 -06:00
Stephen Haberman 6f2194f757 Call removeJob instead of killing the cluster. 2013-01-22 15:38:58 -06:00
Stephen Haberman 27b3f3f0a9 Handle slaveLost before slaveIdToHost knows about it. 2013-01-22 15:30:42 -06:00
Imran Rashid 905c720e5e Merge branch 'master' into blockmanager_ui 
Conflicts:
	core/src/main/scala/spark/RDD.scala
 2013-01-22 12:02:27 -08:00
Matei Zaharia d1de9d7d15 Merge pull request #400 from stephenh/sparkcontextcstrs 
Use default arguments instead of constructor overloads.
 2013-01-22 11:56:54 -08:00
Imran Rashid 50e2b23927 Fix up some problems from the merge 2013-01-22 11:46:01 -08:00
Stephen Haberman 588b24197a Use default arguments instead of constructor overloads. 2013-01-22 10:19:30 -06:00
Leemoonsoo 7e9ee2e833 Fix for hanging spark.HttpFileServer with kind of virtual network 2013-01-22 23:08:34 +09:00
Charles Reiss e353886a8c Use generation numbers for fetch failure tracking 2013-01-22 00:23:31 -08:00
Josh Rosen 551a47a620 Refactor daemon thread pool creation. 2013-01-21 23:31:00 -08:00
Josh Rosen c75ae3622e Make AccumulatorParam an abstract base class. 2013-01-21 22:32:57 -08:00
Stephen Haberman a8baeb9327 Further simplify getOrElse call. 2013-01-21 21:30:24 -06:00
Matei Zaharia 7d3e359f2c Merge pull request #395 from stephenh/fixrecursion 
Remove unneeded/now-broken saveAsNewAPIHadoopFile overload.
 2013-01-21 18:11:31 -08:00
Stephen Haberman 2d8218b871 Remove unneeded/now-broken saveAsNewAPIHadoopFile overload. 2013-01-21 20:00:27 -06:00
Josh Rosen 7b9e96c992 Add synchronization to Executor.updateDependencies() (SPARK-662) 2013-01-21 17:34:23 -08:00
Josh Rosen ef711902c1 Don't download files to master's working directory. 
This should avoid exceptions caused by existing
files with different contents.

I also removed some unused code.
 2013-01-21 17:34:17 -08:00
Matei Zaharia 506077c993 Merge pull request #392 from stephenh/javaapisuite 
Move JavaAPISuite into spark.streaming.
 2013-01-21 15:07:43 -08:00
Stephen Haberman e5ca241335 Move JavaAPISuite into spark.streaming. 2013-01-21 16:06:58 -06:00
Stephen Haberman ffd1623595 Minor cleanup. 2013-01-21 15:55:46 -06:00
Matei Zaharia 76d7c0ce2b Add more Akka settings to docs 2013-01-21 13:10:33 -08:00
Matei Zaharia 2173f6c7ca Clarify the documentation on env variables for standalone mode 2013-01-21 13:03:20 -08:00
Matei Zaharia a88b44ed3b Only bind to IPv4 addresses when trying to auto-detect external IP 2013-01-21 11:59:21 -08:00
Matei Zaharia 4d34c7fc3e Fix compile error caused by cherry-pick 2013-01-21 11:33:48 -08:00
Imran Rashid a3f571b539 more File -> String changes 2013-01-21 11:21:52 -08:00
Imran Rashid fe26acc482 remove unused imports 2013-01-21 11:21:46 -08:00
Imran Rashid c73107500e send sparkHome as String instead of File over network 2013-01-21 11:21:39 -08:00
Imran Rashid 5bf73df7f0 oops, fix stupid compile error 2013-01-21 11:21:33 -08:00
Imran Rashid aae5a920a4 get sparkHome the correct way 2013-01-21 11:21:28 -08:00
Imran Rashid f116d6b5c6 executor can use a different sparkHome from Worker 2013-01-21 11:21:22 -08:00
Matei Zaharia 414b41e1d8 Merge pull request #362 from stephenh/hadoopconf 
Add SparkContext.hadoopConfiguration
 2013-01-21 11:18:48 -08:00