[SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo
In BlockManagermasterActor, _remainingMem would increase memSize for twice when updateBlockInfo if new storageLevel is invalid and old storageLevel is "useMemory". Also, _remainingMem should increase with original memory size instead of new memSize. Author: Zhang, Liye <liye.zhang@intel.com> Closes #2792 from liyezhang556520/spark-3941-remainMem and squashes the following commits: 3d487cc [Zhang, Liye] make the code concise 0380a32 [Zhang, Liye] [SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo
This commit is contained in:
parent
be2ec4a91d
commit
642b246beb
|
@ -457,16 +457,18 @@ private[spark] class BlockManagerInfo(
|
|||
|
||||
if (_blocks.containsKey(blockId)) {
|
||||
// The block exists on the slave already.
|
||||
val originalLevel: StorageLevel = _blocks.get(blockId).storageLevel
|
||||
val blockStatus: BlockStatus = _blocks.get(blockId)
|
||||
val originalLevel: StorageLevel = blockStatus.storageLevel
|
||||
val originalMemSize: Long = blockStatus.memSize
|
||||
|
||||
if (originalLevel.useMemory) {
|
||||
_remainingMem += memSize
|
||||
_remainingMem += originalMemSize
|
||||
}
|
||||
}
|
||||
|
||||
if (storageLevel.isValid) {
|
||||
/* isValid means it is either stored in-memory, on-disk or on-Tachyon.
|
||||
* But the memSize here indicates the data size in or dropped from memory,
|
||||
* The memSize here indicates the data size in or dropped from memory,
|
||||
* tachyonSize here indicates the data size in or dropped from Tachyon,
|
||||
* and the diskSize here indicates the data size in or dropped to disk.
|
||||
* They can be both larger than 0, when a block is dropped from memory to disk.
|
||||
|
@ -493,7 +495,6 @@ private[spark] class BlockManagerInfo(
|
|||
val blockStatus: BlockStatus = _blocks.get(blockId)
|
||||
_blocks.remove(blockId)
|
||||
if (blockStatus.storageLevel.useMemory) {
|
||||
_remainingMem += blockStatus.memSize
|
||||
logInfo("Removed %s on %s in memory (size: %s, free: %s)".format(
|
||||
blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.memSize),
|
||||
Utils.bytesToString(_remainingMem)))
|
||||
|
|
Loading…
Reference in a new issue