[SPARK-29844][ML] Improper unpersist strategy in ml.recommendation.ASL.train
### What changes were proposed in this pull request? Adjust improper unpersist timing on RDD. ### Why are the changes needed? Improper unpersist timing will result in memory waste ### Does this PR introduce any user-facing change? No ### How was this patch tested? Manually Closes #26469 from Icysandwich/SPARK-29844. Authored-by: DongWang <cqwd123@gmail.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
This commit is contained in:
parent
030e5d987e
commit
9a96a20a69
|
@ -1046,13 +1046,13 @@ object ALS extends DefaultParamsReadable[ALS] with Logging {
|
|||
.persist(finalRDDStorageLevel)
|
||||
if (finalRDDStorageLevel != StorageLevel.NONE) {
|
||||
userIdAndFactors.count()
|
||||
itemIdAndFactors.count()
|
||||
itemFactors.unpersist()
|
||||
userInBlocks.unpersist()
|
||||
userOutBlocks.unpersist()
|
||||
itemInBlocks.unpersist()
|
||||
itemOutBlocks.unpersist()
|
||||
blockRatings.unpersist()
|
||||
itemIdAndFactors.count()
|
||||
itemFactors.unpersist()
|
||||
itemInBlocks.unpersist()
|
||||
}
|
||||
(userIdAndFactors, itemIdAndFactors)
|
||||
}
|
||||
|
|
Loading…
Reference in a new issue