[SPARK-23604][SQL] Change Statistics.isEmpty to !Statistics.hasNonNul…

…lValue

## What changes were proposed in this pull request?

Parquet 1.9 will change the semantics of Statistics.isEmpty slightly
to reflect if the null value count has been set. That breaks a
timestamp interoperability test that cares only about whether there
are column values present in the statistics of a written file for an
INT96 column. Fix by using Statistics.hasNonNullValue instead.

## How was this patch tested?

Unit tests continue to pass against Parquet 1.8, and also pass against
a Parquet build including PARQUET-1217.

Author: Henry Robinson <henry@cloudera.com>

Closes #20740 from henryr/spark-23604.
This commit is contained in:
Henry Robinson 2018-03-05 16:49:24 -08:00 committed by Marcelo Vanzin
parent f6b49f9d1b
commit 8c5b34c425

View file

@ -184,7 +184,7 @@ class ParquetInteroperabilitySuite extends ParquetCompatibilityTest with SharedS
// when the data is read back as mentioned above, b/c int96 is unsigned. This
// assert makes sure this holds even if we change parquet versions (if eg. there
// were ever statistics even on unsigned columns).
assert(columnStats.isEmpty)
assert(!columnStats.hasNonNullValue)
}
// These queries should return the entire dataset with the conversion applied,