[SPARK-36659][SQL] Promote spark.sql.execution.topKSortFallbackThreshold to a user-facing config
### What changes were proposed in this pull request? Promote spark.sql.execution.topKSortFallbackThreshold to a user-facing config ### Why are the changes needed? spark.sql.execution.topKSortFallbackThreshold now is an internal config hidden from users Integer.MAX_VALUE - 15 as its default. In many real-world cases, if the K is very big, there would be performance issues. It's better to leave this choice to users ### Does this PR introduce _any_ user-facing change? spark.sql.execution.topKSortFallbackThreshold is now user-facing ### How was this patch tested? passing GA Closes #33904 from yaooqinn/SPARK-36659. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
parent
d3e3df17aa
commit
7f1ad7be18
|
@ -2647,7 +2647,6 @@ object SQLConf {
|
|||
|
||||
val TOP_K_SORT_FALLBACK_THRESHOLD =
|
||||
buildConf("spark.sql.execution.topKSortFallbackThreshold")
|
||||
.internal()
|
||||
.doc("In SQL queries with a SORT followed by a LIMIT like " +
|
||||
"'SELECT x FROM t ORDER BY y LIMIT m', if m is under this threshold, do a top-K sort" +
|
||||
" in memory, otherwise do a global sort which spills to disk if necessary.")
|
||||
|
|
Loading…
Reference in a new issue