Matei Zaharia
|
fe5e4812fc
|
Merge pull request #409 from rxin/splitpruningrdd
Added pruntSplits method to RDD.
|
2013-01-23 22:23:22 -08:00 |
|
Reynold Xin
|
c109f29c97
|
Updated PruneDependency to change "split" to "partition".
|
2013-01-23 22:22:03 -08:00 |
|
Reynold Xin
|
eedc542a02
|
Removed pruneSplits method in RDD and renamed SplitsPruningRDD to
PartitionPruningRDD.
|
2013-01-23 22:14:23 -08:00 |
|
Reynold Xin
|
81004b967e
|
Marked prev RDD as transient in SplitsPruningRDD.
|
2013-01-23 21:54:27 -08:00 |
|
Reynold Xin
|
636e912f32
|
Created a PruneDependency to properly assign dependency for
SplitsPruningRDD.
|
2013-01-23 21:21:55 -08:00 |
|
Reynold Xin
|
45cd50d5fe
|
Updated assert == to ===.
|
2013-01-23 16:06:58 -08:00 |
|
Matei Zaharia
|
548856a224
|
Merge remote-tracking branch 'woggling/remove-machines'
Conflicts:
core/src/main/scala/spark/scheduler/DAGScheduler.scala
|
2013-01-23 15:44:17 -08:00 |
|
Reynold Xin
|
c24b3819dd
|
Added an extra assert for split size check.
|
2013-01-23 15:34:59 -08:00 |
|
Reynold Xin
|
eb222b7206
|
Added pruntSplits method to RDD.
|
2013-01-23 15:29:02 -08:00 |
|
Matei Zaharia
|
1dd82743e0
|
Fix compile error due to cherry-pick
|
2013-01-23 13:07:27 -08:00 |
|
Charles Reiss
|
5c7422292e
|
Remove more dead code from test.
|
2013-01-23 12:59:51 -08:00 |
|
Imran Rashid
|
e1985bfa04
|
be sure to set class loader of kryo instances
|
2013-01-23 12:51:09 -08:00 |
|
Charles Reiss
|
be4a115a7e
|
Clarify TODO.
|
2013-01-23 12:48:45 -08:00 |
|
Charles Reiss
|
88b9d240fd
|
Remove dead code in test.
|
2013-01-23 12:40:38 -08:00 |
|
Matei Zaharia
|
1a3aeeca23
|
Merge pull request #407 from woggling/no-cache-tracker
Eliminate CacheTracker
|
2013-01-23 12:28:48 -08:00 |
|
Charles Reiss
|
e1027ca639
|
Actually add CacheManager.
|
2013-01-23 12:22:11 -08:00 |
|
Matei Zaharia
|
4147e1d47b
|
Merge pull request #406 from tdas/master
Changed StorageLevel and BlockManagerId API to prevent duplication in memory
|
2013-01-23 12:18:31 -08:00 |
|
Matei Zaharia
|
4d77d554e1
|
Merge pull request #394 from JoshRosen/add_file_fix
Add SparkFiles.get() API to access files added through addFile().
|
2013-01-23 12:16:30 -08:00 |
|
Josh Rosen
|
ae2ed2947d
|
Allow PySpark's SparkFiles to be used from driver
Fix minor documentation formatting issues.
|
2013-01-23 10:58:50 -08:00 |
|
Tathagata Das
|
79d55700ce
|
One more fix. Made even default constructor of BlockManagerId private to prevent such problems in the future.
|
2013-01-23 01:57:09 -08:00 |
|
Charles Reiss
|
0b506dd2ec
|
Add tests of various node failure scenarios.
|
2013-01-23 01:38:15 -08:00 |
|
Charles Reiss
|
d209b6b764
|
Extra debugging from hostLost()
|
2013-01-23 01:35:14 -08:00 |
|
Charles Reiss
|
9a27062260
|
Force generation increment after shuffle map stage
|
2013-01-23 01:34:44 -08:00 |
|
Tathagata Das
|
155f31398d
|
Made StorageLevel constructor private, and added StorageLevels.create() to the Java API. Updates scala and java programming guides.
|
2013-01-23 01:10:26 -08:00 |
|
Tathagata Das
|
5e11f1e51f
|
Modified StorageLevel API to ensure zero duplicate objects.
|
2013-01-22 23:42:53 -08:00 |
|
Matei Zaharia
|
7c3a1bddb7
|
Merge pull request #405 from JoshRosen/spark-530
Add test for driver hanging on exit (SPARK-530)
|
2013-01-22 23:28:47 -08:00 |
|
Tathagata Das
|
bacade6caf
|
Modified BlockManagerId API to ensure zero duplicate objects. Fixed BlockManagerId testcase in BlockManagerTestSuite.
|
2013-01-22 22:55:26 -08:00 |
|
Josh Rosen
|
43e9ff9596
|
Add test for driver hanging on exit (SPARK-530).
|
2013-01-22 22:47:26 -08:00 |
|
Charles Reiss
|
2849931000
|
Eliminate CacheTracker.
Replaces DAGScheduler's queries of CacheTracker with BlockManagerMaster
queries.
Adds CacheManager to locally coordinate computation of cached RDDs.
|
2013-01-22 22:19:30 -08:00 |
|
Matei Zaharia
|
ebaa8f6519
|
Merge remote-tracking branch 'stephenh/cleanup'
Conflicts:
core/src/main/scala/spark/scheduler/local/LocalScheduler.scala
|
2013-01-22 21:05:45 -08:00 |
|
Matei Zaharia
|
d2d273868b
|
Merge pull request #397 from JoshRosen/refactoring/daemon-threads
Refactor daemon thread creation
|
2013-01-22 21:02:53 -08:00 |
|
Matei Zaharia
|
05f13280de
|
Merge pull request #404 from mbautin/master-fix-avro-dependency-in-repl
Add an Avro dependency to REPL to make it compile with Hadoop 2
|
2013-01-22 19:59:52 -08:00 |
|
Mikhail Bautin
|
325297e5c3
|
Add an Avro dependency to REPL to make it compile with Hadoop 2
|
2013-01-22 18:11:51 -08:00 |
|
Josh Rosen
|
35168d9c89
|
Fix sys.path bug in PySpark SparkContext.addPyFile
|
2013-01-22 17:54:11 -08:00 |
|
Matei Zaharia
|
151c47eef5
|
Merge pull request #399 from NFLabs/master
Fix for hanging spark.HttpFileServer on the kind of virtual network
|
2013-01-22 15:49:24 -08:00 |
|
Matei Zaharia
|
49c78446c8
|
Merge pull request #402 from stephenh/handleslavelosttoosoon
Handle slaveLost before slaveIdToHost knows about it.
|
2013-01-22 15:48:37 -08:00 |
|
Stephen Haberman
|
27b3f3f0a9
|
Handle slaveLost before slaveIdToHost knows about it.
|
2013-01-22 15:30:42 -06:00 |
|
Matei Zaharia
|
d1de9d7d15
|
Merge pull request #400 from stephenh/sparkcontextcstrs
Use default arguments instead of constructor overloads.
|
2013-01-22 11:56:54 -08:00 |
|
Stephen Haberman
|
588b24197a
|
Use default arguments instead of constructor overloads.
|
2013-01-22 10:19:30 -06:00 |
|
Leemoonsoo
|
7e9ee2e833
|
Fix for hanging spark.HttpFileServer with kind of virtual network
|
2013-01-22 23:08:34 +09:00 |
|
Charles Reiss
|
e353886a8c
|
Use generation numbers for fetch failure tracking
|
2013-01-22 00:23:31 -08:00 |
|
Josh Rosen
|
551a47a620
|
Refactor daemon thread pool creation.
|
2013-01-21 23:31:00 -08:00 |
|
Stephen Haberman
|
a8baeb9327
|
Further simplify getOrElse call.
|
2013-01-21 21:30:24 -06:00 |
|
Matei Zaharia
|
7d3e359f2c
|
Merge pull request #395 from stephenh/fixrecursion
Remove unneeded/now-broken saveAsNewAPIHadoopFile overload.
|
2013-01-21 18:11:31 -08:00 |
|
Stephen Haberman
|
2d8218b871
|
Remove unneeded/now-broken saveAsNewAPIHadoopFile overload.
|
2013-01-21 20:00:27 -06:00 |
|
Josh Rosen
|
7b9e96c992
|
Add synchronization to Executor.updateDependencies() (SPARK-662)
|
2013-01-21 17:34:23 -08:00 |
|
Josh Rosen
|
ef711902c1
|
Don't download files to master's working directory.
This should avoid exceptions caused by existing
files with different contents.
I also removed some unused code.
|
2013-01-21 17:34:17 -08:00 |
|
Matei Zaharia
|
506077c993
|
Merge pull request #392 from stephenh/javaapisuite
Move JavaAPISuite into spark.streaming.
|
2013-01-21 15:07:43 -08:00 |
|
Stephen Haberman
|
e5ca241335
|
Move JavaAPISuite into spark.streaming.
|
2013-01-21 16:06:58 -06:00 |
|
Stephen Haberman
|
ffd1623595
|
Minor cleanup.
|
2013-01-21 15:55:46 -06:00 |
|