ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-30339][SQL] Avoid to fail twice in function lookup

Browse source 
### What changes were proposed in this pull request?

Currently if function lookup fails, spark will give it a second change by casting decimal type to double type. But for cases where decimal type doesn't exist, it's meaningless to lookup again and causes extra cost like unnecessary metastore access. We should throw exceptions directly in these cases.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

Covered by existing tests.

Closes #26994 from wzhfy/avoid_udf_fail_twice.

Authored-by: Zhenhua Wang <wzh_zju@163.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
This commit is contained in:
Zhenhua Wang 2019-12-31 01:09:51 +09:00 committed by HyukjinKwon
parent e054a0af6f
commit a8bf5d823b
2 changed files with 1 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala
View file

@ -117,7 +117,7 @@ private[sql] class HiveSessionCatalog(
try {
lookupFunction0(name, children)
} catch {
case NonFatal(_) =>
case NonFatal(_) if children.exists(_.dataType.isInstanceOf[DecimalType]) =>
// SPARK-16228 ExternalCatalog may recognize `double`-type only.
val newChildren = children.map { child =>
if (child.dataType.isInstanceOf[DecimalType]) Cast(child, DoubleType) else child

7
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
View file

@ -149,13 +149,6 @@ class HiveUDFSuite extends QueryTest with TestHiveSingleton with SQLTestUtils {
sql("SELECT array(max(key), max(key)) FROM src").collect().toSeq)
}
test("SPARK-16228 Percentile needs explicit cast to double") {
sql("select percentile(value, cast(0.5 as double)) from values 1,2,3 T(value)")
sql("select percentile_approx(value, cast(0.5 as double)) from values 1.0,2.0,3.0 T(value)")
sql("select percentile(value, 0.5) from values 1,2,3 T(value)")
sql("select percentile_approx(value, 0.5) from values 1.0,2.0,3.0 T(value)")
}
test("Generic UDAF aggregates") {
checkAnswer(sql(