[SPARK-29844][ML] Improper unpersist strategy in ml.recommendation.ASL.train

### What changes were proposed in this pull request?
Adjust improper unpersist timing on RDD.

### Why are the changes needed?
Improper unpersist timing will result in memory waste

### Does this PR introduce any user-facing change?
No

### How was this patch tested?
Manually

Closes #26469 from Icysandwich/SPARK-29844.

Authored-by: DongWang <cqwd123@gmail.com>
Signed-off-by: Sean Owen <sean.owen@databricks.com>
This commit is contained in:
DongWang 2019-11-12 18:31:58 -06:00 committed by Sean Owen
parent 030e5d987e
commit 9a96a20a69

View file

@ -1046,13 +1046,13 @@ object ALS extends DefaultParamsReadable[ALS] with Logging {
.persist(finalRDDStorageLevel)
if (finalRDDStorageLevel != StorageLevel.NONE) {
userIdAndFactors.count()
itemIdAndFactors.count()
itemFactors.unpersist()
userInBlocks.unpersist()
userOutBlocks.unpersist()
itemInBlocks.unpersist()
itemOutBlocks.unpersist()
blockRatings.unpersist()
itemIdAndFactors.count()
itemFactors.unpersist()
itemInBlocks.unpersist()
}
(userIdAndFactors, itemIdAndFactors)
}