spark-instrumented-optimizer/sql/core
maryannxue b4fd75fb9b [SPARK-24972][SQL] PivotFirst could not handle pivot columns of complex types
## What changes were proposed in this pull request?

When the pivot column is of a complex type, the eval() result will be an UnsafeRow, while the keys of the HashMap for column value matching is a GenericInternalRow. As a result, there will be no match and the result will always be empty.
So for a pivot column of complex-types, we should:
1) If the complex-type is not comparable (orderable), throw an Exception. It cannot be a pivot column.
2) Otherwise, if it goes through the `PivotFirst` code path, `PivotFirst` should use a TreeMap instead of HashMap for such columns.

This PR has also reverted the walk-around in Analyzer that had been introduced to avoid this `PivotFirst` issue.

## How was this patch tested?

Added UT.

Author: maryannxue <maryannxue@apache.org>

Closes #21926 from maryannxue/pivot_followup.
2018-07-30 23:43:53 -07:00
..
benchmarks [SPARK-24549][SQL] Support Decimal type push down to the parquet data sources 2018-07-16 15:44:51 +08:00
src [SPARK-24972][SQL] PivotFirst could not handle pivot columns of complex types 2018-07-30 23:43:53 -07:00
pom.xml [SPARK-24771][BUILD] Upgrade Apache AVRO to 1.8.2 2018-07-30 07:30:47 -07:00