[SPARK-23604][SQL] Change Statistics.isEmpty to !Statistics.hasNonNul…
…lValue ## What changes were proposed in this pull request? Parquet 1.9 will change the semantics of Statistics.isEmpty slightly to reflect if the null value count has been set. That breaks a timestamp interoperability test that cares only about whether there are column values present in the statistics of a written file for an INT96 column. Fix by using Statistics.hasNonNullValue instead. ## How was this patch tested? Unit tests continue to pass against Parquet 1.8, and also pass against a Parquet build including PARQUET-1217. Author: Henry Robinson <henry@cloudera.com> Closes #20740 from henryr/spark-23604.
This commit is contained in:
parent
f6b49f9d1b
commit
8c5b34c425
|
@ -184,7 +184,7 @@ class ParquetInteroperabilitySuite extends ParquetCompatibilityTest with SharedS
|
|||
// when the data is read back as mentioned above, b/c int96 is unsigned. This
|
||||
// assert makes sure this holds even if we change parquet versions (if eg. there
|
||||
// were ever statistics even on unsigned columns).
|
||||
assert(columnStats.isEmpty)
|
||||
assert(!columnStats.hasNonNullValue)
|
||||
}
|
||||
|
||||
// These queries should return the entire dataset with the conversion applied,
|
||||
|
|
Loading…
Reference in a new issue