Charles Reiss
5c7422292e
Remove more dead code from test.
2013-01-23 12:59:51 -08:00
Imran Rashid
e1985bfa04
be sure to set class loader of kryo instances
2013-01-23 12:51:09 -08:00
Charles Reiss
be4a115a7e
Clarify TODO.
2013-01-23 12:48:45 -08:00
Charles Reiss
88b9d240fd
Remove dead code in test.
2013-01-23 12:40:38 -08:00
Matei Zaharia
1a3aeeca23
Merge pull request #407 from woggling/no-cache-tracker
...
Eliminate CacheTracker
2013-01-23 12:28:48 -08:00
Charles Reiss
e1027ca639
Actually add CacheManager.
2013-01-23 12:22:11 -08:00
Matei Zaharia
4147e1d47b
Merge pull request #406 from tdas/master
...
Changed StorageLevel and BlockManagerId API to prevent duplication in memory
2013-01-23 12:18:31 -08:00
Matei Zaharia
4d77d554e1
Merge pull request #394 from JoshRosen/add_file_fix
...
Add SparkFiles.get() API to access files added through addFile().
2013-01-23 12:16:30 -08:00
Josh Rosen
b47d054cfc
Remove use of abc.ABCMeta due to cloudpickle issue.
...
cloudpickle runs into issues while pickling subclasses of AccumulatorParam,
which may be related to this Python issue:
http://bugs.python.org/issue7689
This seems hard to fix and the ABCMeta wasn't necessary, so I removed it.
2013-01-23 11:47:27 -08:00
Josh Rosen
ae2ed2947d
Allow PySpark's SparkFiles to be used from driver
...
Fix minor documentation formatting issues.
2013-01-23 10:58:50 -08:00
Tathagata Das
79d55700ce
One more fix. Made even default constructor of BlockManagerId private to prevent such problems in the future.
2013-01-23 01:57:09 -08:00
Charles Reiss
0b506dd2ec
Add tests of various node failure scenarios.
2013-01-23 01:38:15 -08:00
Charles Reiss
d209b6b764
Extra debugging from hostLost()
2013-01-23 01:35:14 -08:00
Charles Reiss
9a27062260
Force generation increment after shuffle map stage
2013-01-23 01:34:44 -08:00
Tathagata Das
155f31398d
Made StorageLevel constructor private, and added StorageLevels.create() to the Java API. Updates scala and java programming guides.
2013-01-23 01:10:26 -08:00
Tathagata Das
5e11f1e51f
Modified StorageLevel API to ensure zero duplicate objects.
2013-01-22 23:42:53 -08:00
Matei Zaharia
7c3a1bddb7
Merge pull request #405 from JoshRosen/spark-530
...
Add test for driver hanging on exit (SPARK-530)
2013-01-22 23:28:47 -08:00
Tathagata Das
bacade6caf
Modified BlockManagerId API to ensure zero duplicate objects. Fixed BlockManagerId testcase in BlockManagerTestSuite.
2013-01-22 22:55:26 -08:00
Josh Rosen
43e9ff9596
Add test for driver hanging on exit (SPARK-530).
2013-01-22 22:47:26 -08:00
Charles Reiss
2849931000
Eliminate CacheTracker.
...
Replaces DAGScheduler's queries of CacheTracker with BlockManagerMaster
queries.
Adds CacheManager to locally coordinate computation of cached RDDs.
2013-01-22 22:19:30 -08:00
Matei Zaharia
ebaa8f6519
Merge remote-tracking branch 'stephenh/cleanup'
...
Conflicts:
core/src/main/scala/spark/scheduler/local/LocalScheduler.scala
2013-01-22 21:05:45 -08:00
Matei Zaharia
d2d273868b
Merge pull request #397 from JoshRosen/refactoring/daemon-threads
...
Refactor daemon thread creation
2013-01-22 21:02:53 -08:00
Matei Zaharia
05f13280de
Merge pull request #404 from mbautin/master-fix-avro-dependency-in-repl
...
Add an Avro dependency to REPL to make it compile with Hadoop 2
2013-01-22 19:59:52 -08:00
Mikhail Bautin
325297e5c3
Add an Avro dependency to REPL to make it compile with Hadoop 2
2013-01-22 18:11:51 -08:00
Josh Rosen
35168d9c89
Fix sys.path bug in PySpark SparkContext.addPyFile
2013-01-22 17:54:11 -08:00
Stephen Haberman
98d0b7747d
Fix Worker logInfo about unknown executor.
2013-01-22 18:11:51 -06:00
Stephen Haberman
8c51322cd0
Don't bother creating an exception.
2013-01-22 18:09:10 -06:00
Stephen Haberman
fdec42385a
Fix SPARK_MEM in ExecutorRunner.
2013-01-22 18:01:12 -06:00
Stephen Haberman
2437f6741b
Restore SPARK_MEM in executorEnvs.
2013-01-22 18:01:03 -06:00
Matei Zaharia
151c47eef5
Merge pull request #399 from NFLabs/master
...
Fix for hanging spark.HttpFileServer on the kind of virtual network
2013-01-22 15:49:24 -08:00
Matei Zaharia
49c78446c8
Merge pull request #402 from stephenh/handleslavelosttoosoon
...
Handle slaveLost before slaveIdToHost knows about it.
2013-01-22 15:48:37 -08:00
Stephen Haberman
250fe89679
Handle Master telling the Worker to kill an already-dead executor.
2013-01-22 16:29:05 -06:00
Stephen Haberman
6f2194f757
Call removeJob instead of killing the cluster.
2013-01-22 15:38:58 -06:00
Stephen Haberman
27b3f3f0a9
Handle slaveLost before slaveIdToHost knows about it.
2013-01-22 15:30:42 -06:00
Imran Rashid
905c720e5e
Merge branch 'master' into blockmanager_ui
...
Conflicts:
core/src/main/scala/spark/RDD.scala
2013-01-22 12:02:27 -08:00
Matei Zaharia
d1de9d7d15
Merge pull request #400 from stephenh/sparkcontextcstrs
...
Use default arguments instead of constructor overloads.
2013-01-22 11:56:54 -08:00
Imran Rashid
50e2b23927
Fix up some problems from the merge
2013-01-22 11:46:01 -08:00
Stephen Haberman
588b24197a
Use default arguments instead of constructor overloads.
2013-01-22 10:19:30 -06:00
Leemoonsoo
7e9ee2e833
Fix for hanging spark.HttpFileServer with kind of virtual network
2013-01-22 23:08:34 +09:00
Charles Reiss
e353886a8c
Use generation numbers for fetch failure tracking
2013-01-22 00:23:31 -08:00
Josh Rosen
551a47a620
Refactor daemon thread pool creation.
2013-01-21 23:31:00 -08:00
Josh Rosen
c75ae3622e
Make AccumulatorParam an abstract base class.
2013-01-21 22:32:57 -08:00
Stephen Haberman
a8baeb9327
Further simplify getOrElse call.
2013-01-21 21:30:24 -06:00
Matei Zaharia
7d3e359f2c
Merge pull request #395 from stephenh/fixrecursion
...
Remove unneeded/now-broken saveAsNewAPIHadoopFile overload.
2013-01-21 18:11:31 -08:00
Stephen Haberman
2d8218b871
Remove unneeded/now-broken saveAsNewAPIHadoopFile overload.
2013-01-21 20:00:27 -06:00
Josh Rosen
7b9e96c992
Add synchronization to Executor.updateDependencies() (SPARK-662)
2013-01-21 17:34:23 -08:00
Josh Rosen
ef711902c1
Don't download files to master's working directory.
...
This should avoid exceptions caused by existing
files with different contents.
I also removed some unused code.
2013-01-21 17:34:17 -08:00
Matei Zaharia
506077c993
Merge pull request #392 from stephenh/javaapisuite
...
Move JavaAPISuite into spark.streaming.
2013-01-21 15:07:43 -08:00
Stephen Haberman
e5ca241335
Move JavaAPISuite into spark.streaming.
2013-01-21 16:06:58 -06:00
Stephen Haberman
ffd1623595
Minor cleanup.
2013-01-21 15:55:46 -06:00