[SPARK-15300] Fix writer lock conflict when remove a block
## What changes were proposed in this pull request? A writer lock could be acquired when 1) create a new block 2) remove a block 3) evict a block to disk. 1) and 3) could happen in the same time within the same task, all of them could happen in the same time outside a task. It's OK that when someone try to grab the write block for a block, but the block is acquired by another one that has the same task attempt id. This PR remove the check. ## How was this patch tested? Updated existing tests. Author: Davies Liu <davies@databricks.com> Closes #13082 from davies/write_lock_conflict.
This commit is contained in:
parent
ef7a5e0bca
commit
ad182086cc
|
@ -228,10 +228,7 @@ private[storage] class BlockInfoManager extends Logging {
|
|||
infos.get(blockId) match {
|
||||
case None => return None
|
||||
case Some(info) =>
|
||||
if (info.writerTask == currentTaskAttemptId) {
|
||||
throw new IllegalStateException(
|
||||
s"Task $currentTaskAttemptId has already locked $blockId for writing")
|
||||
} else if (info.writerTask == BlockInfo.NO_WRITER && info.readerCount == 0) {
|
||||
if (info.writerTask == BlockInfo.NO_WRITER && info.readerCount == 0) {
|
||||
info.writerTask = currentTaskAttemptId
|
||||
writeLocksByTask.addBinding(currentTaskAttemptId, blockId)
|
||||
logTrace(s"Task $currentTaskAttemptId acquired write lock for $blockId")
|
||||
|
|
|
@ -208,16 +208,14 @@ class BlockInfoManagerSuite extends SparkFunSuite with BeforeAndAfterEach {
|
|||
}
|
||||
}
|
||||
|
||||
test("cannot call lockForWriting while already holding a write lock") {
|
||||
test("cannot grab a writer lock while already holding a write lock") {
|
||||
withTaskId(0) {
|
||||
assert(blockInfoManager.lockNewBlockForWriting("block", newBlockInfo()))
|
||||
blockInfoManager.unlock("block")
|
||||
}
|
||||
withTaskId(1) {
|
||||
assert(blockInfoManager.lockForWriting("block").isDefined)
|
||||
intercept[IllegalStateException] {
|
||||
blockInfoManager.lockForWriting("block")
|
||||
}
|
||||
assert(blockInfoManager.lockForWriting("block", false).isEmpty)
|
||||
blockInfoManager.assertBlockIsLockedForWriting("block")
|
||||
}
|
||||
}
|
||||
|
|
Loading…
Reference in a new issue