ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-30067][CORE] Fix fragment offset comparison in getBlockHosts

Browse source 
### What changes were proposed in this pull request?

A bug fixed about the code in getBlockHosts() function. In the case "The fragment ends at a position within this block", the end of fragment should be before the end of block，where the "end of block" means `b.getOffset + b.getLength`，not `b.getLength`.

### Why are the changes needed?

When comparing the fragment end and the block end，we should use fragment's `offset + length`，and then compare to the block's `b.getOffset + b.getLength`, not the block's length.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?
No test.

Closes #26650 from mdianjun/fix-getBlockHosts.

Authored-by: madianjun <madianjun@jd.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
This commit is contained in:
madianjun 2019-12-05 23:39:49 -08:00 committed by Dongjoon Hyun
parent da27f91560
commit a5ccbced8c
1 changed files with 4 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
sql/core/src/main/scala/org/apache/spark/sql/execution/PartitionedFileUtil.scala
View file

@ -64,13 +64,14 @@ object PartitionedFileUtil {
offset: Long,
length: Long): Array[String] = {
val candidates = blockLocations.map {
// The fragment starts from a position within this block
// The fragment starts from a position within this block. It handles the case where the
// fragment is fully contained in the block.
case b if b.getOffset <= offset && offset < b.getOffset + b.getLength =>
b.getHosts -> (b.getOffset + b.getLength - offset).min(length)
// The fragment ends at a position within this block
case b if offset <= b.getOffset && offset + length < b.getLength =>
b.getHosts -> (offset + length - b.getOffset).min(length)
case b if b.getOffset < offset + length && offset + length < b.getOffset + b.getLength =>
b.getHosts -> (offset + length - b.getOffset)
// The fragment fully contains this block
case b if offset <= b.getOffset && b.getOffset + b.getLength <= offset + length =>