spark-instrumented-optimizer/python/pyspark/sql
Franklyn D'souza 0f90f4e6ac [SPARK-13410][SQL] Support unionAll for DataFrames with UDT columns.
## What changes were proposed in this pull request?

This PR adds equality operators to UDT classes so that they can be correctly tested for dataType equality during union operations.

This was previously causing `"AnalysisException: u"unresolved operator 'Union;""` when trying to unionAll two dataframes with UDT columns as below.

```
from pyspark.sql.tests import PythonOnlyPoint, PythonOnlyUDT
from pyspark.sql import types

schema = types.StructType([types.StructField("point", PythonOnlyUDT(), True)])

a = sqlCtx.createDataFrame([[PythonOnlyPoint(1.0, 2.0)]], schema)
b = sqlCtx.createDataFrame([[PythonOnlyPoint(3.0, 4.0)]], schema)

c = a.unionAll(b)
```

## How was the this patch tested?

Tested using two unit tests in sql/test.py and the DataFrameSuite.

Additional information here : https://issues.apache.org/jira/browse/SPARK-13410

Author: Franklyn D'souza <franklynd@gmail.com>

Closes #11279 from damnMeddlingKid/udt-union-all.
2016-02-21 16:58:17 -08:00
..
__init__.py [SPARK-12600][SQL] Remove deprecated methods in Spark SQL 2016-01-04 18:02:38 -08:00
column.py [SPARK-12799] Simplify various string output for expressions 2016-02-21 22:53:15 +08:00
context.py [SPARK-12799] Simplify various string output for expressions 2016-02-21 22:53:15 +08:00
dataframe.py [SPARK-13296][SQL] Move UserDefinedFunction into sql.expressions. 2016-02-13 21:06:31 -08:00
functions.py [SPARK-12799] Simplify various string output for expressions 2016-02-21 22:53:15 +08:00
group.py [SPARK-12756][SQL] use hash expression in Exchange 2016-01-13 22:43:28 -08:00
readwriter.py [SPARK-12749][SQL] add json option to parse floating-point types as DecimalType 2016-01-28 15:25:57 -08:00
tests.py [SPARK-13410][SQL] Support unionAll for DataFrames with UDT columns. 2016-02-21 16:58:17 -08:00
types.py [SPARK-11158][SQL] Modified _verify_type() to be more informative on Errors by presenting the Object 2015-10-18 11:39:19 -07:00
utils.py [SPARK-11804] [PYSPARK] Exception raise when using Jdbc predicates opt… 2015-11-18 08:18:54 -08:00
window.py [SPARK-10373] [PYSPARK] move @since into pyspark from sql 2015-09-08 20:56:22 -07:00