[SPARK-30067][CORE] Fix fragment offset comparison in getBlockHosts
### What changes were proposed in this pull request? A bug fixed about the code in getBlockHosts() function. In the case "The fragment ends at a position within this block", the end of fragment should be before the end of block,where the "end of block" means `b.getOffset + b.getLength`,not `b.getLength`. ### Why are the changes needed? When comparing the fragment end and the block end,we should use fragment's `offset + length`,and then compare to the block's `b.getOffset + b.getLength`, not the block's length. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? No test. Closes #26650 from mdianjun/fix-getBlockHosts. Authored-by: madianjun <madianjun@jd.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
parent
da27f91560
commit
a5ccbced8c
|
@ -64,13 +64,14 @@ object PartitionedFileUtil {
|
|||
offset: Long,
|
||||
length: Long): Array[String] = {
|
||||
val candidates = blockLocations.map {
|
||||
// The fragment starts from a position within this block
|
||||
// The fragment starts from a position within this block. It handles the case where the
|
||||
// fragment is fully contained in the block.
|
||||
case b if b.getOffset <= offset && offset < b.getOffset + b.getLength =>
|
||||
b.getHosts -> (b.getOffset + b.getLength - offset).min(length)
|
||||
|
||||
// The fragment ends at a position within this block
|
||||
case b if offset <= b.getOffset && offset + length < b.getLength =>
|
||||
b.getHosts -> (offset + length - b.getOffset).min(length)
|
||||
case b if b.getOffset < offset + length && offset + length < b.getOffset + b.getLength =>
|
||||
b.getHosts -> (offset + length - b.getOffset)
|
||||
|
||||
// The fragment fully contains this block
|
||||
case b if offset <= b.getOffset && b.getOffset + b.getLength <= offset + length =>
|
||||
|
|
Loading…
Reference in a new issue