spark-instrumented-optimizer/python/pyspark/sql
Bryan Cutler e599837248 [SPARK-23009][PYTHON] Fix for non-str col names to createDataFrame from Pandas
## What changes were proposed in this pull request?

This the case when calling `SparkSession.createDataFrame` using a Pandas DataFrame that has non-str column labels.

The column name conversion logic to handle non-string or unicode in python2 is:
```
if column is not any type of string:
    name = str(column)
else if column is unicode in Python 2:
    name = column.encode('utf-8')
```

## How was this patch tested?

Added a new test with a Pandas DataFrame that has int column labels

Author: Bryan Cutler <cutlerb@gmail.com>

Closes #20210 from BryanCutler/python-createDataFrame-int-col-error-SPARK-23009.
2018-01-10 14:55:24 +09:00
..
__init__.py [SPARK-22369][PYTHON][DOCS] Exposes catalog API documentation in PySpark 2017-11-02 15:22:52 +01:00
catalog.py [SPARK-22939][PYSPARK] Support Spark UDF in registerFunction 2018-01-04 21:07:31 +08:00
column.py [SPARK-19165][PYTHON][SQL] PySpark APIs using columns as arguments should validate input types for column 2017-08-24 20:29:03 +09:00
conf.py [SPARK-15464][ML][MLLIB][SQL][TESTS] Replace SQLContext and SparkContext with SparkSession using builder pattern in python test code 2016-05-23 18:14:48 -07:00
context.py [SPARK-22939][PYSPARK] Support Spark UDF in registerFunction 2018-01-04 21:07:31 +08:00
dataframe.py [SPARK-22874][PYSPARK][SQL] Modify checking pandas version to use LooseVersion. 2017-12-22 20:09:51 +09:00
functions.py [SPARK-22930][PYTHON][SQL] Improve the description of Vectorized UDFs for non-deterministic cases 2018-01-06 16:11:20 +08:00
group.py [SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0 2017-12-21 20:43:56 +09:00
readwriter.py [SPARK-22818][SQL] csv escape of quote escape 2017-12-29 07:30:06 +08:00
session.py [SPARK-23009][PYTHON] Fix for non-str col names to createDataFrame from Pandas 2018-01-10 14:55:24 +09:00
streaming.py [SPARK-22933][SPARKR] R Structured Streaming API for withWatermark, trigger, partitionBy 2018-01-03 21:43:14 -08:00
tests.py [SPARK-23009][PYTHON] Fix for non-str col names to createDataFrame from Pandas 2018-01-10 14:55:24 +09:00
types.py [SPARK-22566][PYTHON] Better error message for _merge_type in Pandas to Spark DF conversion 2018-01-08 14:32:05 +09:00
udf.py [SPARK-22901][PYTHON][FOLLOWUP] Adds the doc for asNondeterministic for wrapped UDF function 2018-01-06 23:08:26 +08:00
utils.py [SPARK-22874][PYSPARK][SQL][FOLLOW-UP] Modify error messages to show actual versions. 2017-12-25 20:29:10 +09:00
window.py [SPARK-18690][PYTHON][SQL] Backward compatibility of unbounded frames 2016-12-02 17:39:28 -08:00