ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-5831][Streaming]When checkpoint file size is bigger than 10, then delete the old ones

Browse source 
Author: Xutingjun <1039320815@qq.com>

Closes #4621 from XuTingjun/checkpointFile and squashes the following commits:

b5f2816 [Xutingjun] fix bug
This commit is contained in:
Xutingjun 2015-02-16 14:54:23 +00:00 committed by Sean Owen
parent 3ce58cf9c0
commit 1115e8e739
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
View file

@ -152,7 +152,7 @@ class CheckpointWriter(
// Delete old checkpoint files // Delete old checkpoint files
val allCheckpointFiles = Checkpoint.getCheckpointFiles(checkpointDir, fs) val allCheckpointFiles = Checkpoint.getCheckpointFiles(checkpointDir, fs)
if (allCheckpointFiles.size > 4) { if (allCheckpointFiles.size > 10) {
allCheckpointFiles.take(allCheckpointFiles.size - 10).foreach(file => { allCheckpointFiles.take(allCheckpointFiles.size - 10).foreach(file => {
logInfo("Deleting " + file) logInfo("Deleting " + file)
fs.delete(file, true) fs.delete(file, true)