ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-35791][SQL] Release on-going map properly for NULL-aware ANTI join

Browse source 
### What changes were proposed in this pull request?

NULL-aware ANTI join (https://issues.apache.org/jira/browse/SPARK-32290) detects NULL join keys during building the map for `HashedRelation`, and will immediately return `HashedRelationWithAllNullKeys` without taking care of the map built already. Before returning `HashedRelationWithAllNullKeys`, the map needs to be freed properly to save memory and keep memory accounting correctly.

### Why are the changes needed?

Save memory and keep memory accounting correctly for the join query.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing unit tests introduced in https://github.com/apache/spark/pull/29104 .

Closes #32939 from c21/free-null-aware.

Authored-by: Cheng Su <chengsu@fb.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
This commit is contained in:
Cheng Su 2021-06-17 13:57:35 +08:00 committed by Wenchen Fan
parent 947c7ea27c
commit e0d81d9b71
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
View file

@ -479,6 +479,7 @@ private[joins] object UnsafeHashedRelation {
throw QueryExecutionErrors.cannotAcquireMemoryToBuildUnsafeHashedRelationError() throw QueryExecutionErrors.cannotAcquireMemoryToBuildUnsafeHashedRelationError()
} }
} else if (isNullAware) { } else if (isNullAware) {
binaryMap.free()
return HashedRelationWithAllNullKeys return HashedRelationWithAllNullKeys
} }
} }
@ -1060,6 +1061,7 @@ private[joins] object LongHashedRelation {
val key = rowKey.getLong(0) val key = rowKey.getLong(0)
map.append(key, unsafeRow) map.append(key, unsafeRow)
} else if (isNullAware) { } else if (isNullAware) {
map.free()
return HashedRelationWithAllNullKeys return HashedRelationWithAllNullKeys
} }
} }