Angerszhuuuu 2ff038a7b3 [SPARK-36753][SQL] ArrayExcept handle duplicated Double.NaN and Float.NaN ... ### What changes were proposed in this pull request? For query ``` select array_except(array(cast('nan' as double), 1d), array(cast('nan' as double))) ``` This returns [NaN, 1d], but it should return [1d]. This issue is caused by `OpenHashSet` can't handle `Double.NaN` and `Float.NaN` too. In this pr fix this based on https://github.com/apache/spark/pull/33955 ### Why are the changes needed? Fix bug ### Does this PR introduce _any_ user-facing change? ArrayExcept won't show handle equal `NaN` value ### How was this patch tested? Added UT Closes #33994 from AngersZhuuuu/SPARK-36753. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit a7cbe69986) Signed-off-by: Wenchen Fan <wenchen@databricks.com>		2021-09-22 23:51:58 +08:00
..
benchmarks	[SPARK-34950][TESTS] Update benchmark results to the ones created by GitHub Actions machines	2021-04-03 23:02:56 +03:00
src	[SPARK-36753][SQL] ArrayExcept handle duplicated Double.NaN and Float.NaN	2021-09-22 23:51:58 +08:00
pom.xml	Preparing development version 3.2.1-SNAPSHOT	2021-09-18 11:30:12 +00:00