[SPARK-20637][CORE] Remove mention of old RDD classes from comments
## What changes were proposed in this pull request? A few comments around the code mention RDD classes that do not exist anymore. I'm not sure of the best way to replace these, so I've just removed them here. ## How was this patch tested? Only changes code comments, no testing required Author: Michael Mior <mmior@uwaterloo.ca> Closes #17900 from michaelmior/remove-old-rdds.
This commit is contained in:
parent
ca4625e0e5
commit
a4cbf26bca
|
@ -58,7 +58,7 @@ import org.apache.spark.util._
|
|||
* set of map output files, and another to read those files after a barrier). In the end, every
|
||||
* stage will have only shuffle dependencies on other stages, and may compute multiple operations
|
||||
* inside it. The actual pipelining of these operations happens in the RDD.compute() functions of
|
||||
* various RDDs (MappedRDD, FilteredRDD, etc).
|
||||
* various RDDs
|
||||
*
|
||||
* In addition to coming up with a DAG of stages, the DAGScheduler also determines the preferred
|
||||
* locations to run each task on, based on the current cache status, and passes these to the
|
||||
|
|
|
@ -114,7 +114,7 @@ trait RDDCheckpointTester { self: SparkFunSuite =>
|
|||
* RDDs partitions. So even if the parent RDD is checkpointed and its partitions changed,
|
||||
* the generated RDD will remember the partitions and therefore potentially the whole lineage.
|
||||
* This function should be called only those RDD whose partitions refer to parent RDD's
|
||||
* partitions (i.e., do not call it on simple RDD like MappedRDD).
|
||||
* partitions (i.e., do not call it on simple RDDs).
|
||||
*
|
||||
* @param op an operation to run on the RDD
|
||||
* @param reliableCheckpoint if true, use reliable checkpoints, otherwise use local checkpoints
|
||||
|
@ -388,7 +388,7 @@ class CheckpointSuite extends SparkFunSuite with RDDCheckpointTester with LocalS
|
|||
// the parent RDD has been checkpointed and parent partitions have been changed.
|
||||
// Note that this test is very specific to the current implementation of CartesianRDD.
|
||||
val ones = sc.makeRDD(1 to 100, 10).map(x => x)
|
||||
checkpoint(ones, reliableCheckpoint) // checkpoint that MappedRDD
|
||||
checkpoint(ones, reliableCheckpoint)
|
||||
val cartesian = new CartesianRDD(sc, ones, ones)
|
||||
val splitBeforeCheckpoint =
|
||||
serializeDeserialize(cartesian.partitions.head.asInstanceOf[CartesianPartition])
|
||||
|
@ -411,7 +411,7 @@ class CheckpointSuite extends SparkFunSuite with RDDCheckpointTester with LocalS
|
|||
// Note that this test is very specific to the current implementation of
|
||||
// CoalescedRDDPartitions.
|
||||
val ones = sc.makeRDD(1 to 100, 10).map(x => x)
|
||||
checkpoint(ones, reliableCheckpoint) // checkpoint that MappedRDD
|
||||
checkpoint(ones, reliableCheckpoint)
|
||||
val coalesced = new CoalescedRDD(ones, 2)
|
||||
val splitBeforeCheckpoint =
|
||||
serializeDeserialize(coalesced.partitions.head.asInstanceOf[CoalescedRDDPartition])
|
||||
|
|
|
@ -184,7 +184,7 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkContext with Match
|
|||
sc.listenerBus.waitUntilEmpty(WAIT_TIMEOUT_MILLIS)
|
||||
listener.stageInfos.size should be {1}
|
||||
val stageInfo2 = listener.stageInfos.keys.find(_.stageId == 1).get
|
||||
stageInfo2.rddInfos.size should be {3} // ParallelCollectionRDD, FilteredRDD, MappedRDD
|
||||
stageInfo2.rddInfos.size should be {3}
|
||||
stageInfo2.rddInfos.forall(_.numPartitions == 4) should be {true}
|
||||
stageInfo2.rddInfos.exists(_.name == "Deux") should be {true}
|
||||
listener.stageInfos.clear()
|
||||
|
|
Loading…
Reference in a new issue