ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/python/pyspark/sql
History
Maxim Gekk 64ad7b841d [SPARK-23772][FOLLOW-UP][SQL] Provide an option to ignore column of all null values or empty array during JSON schema inference 
## What changes were proposed in this pull request?

The `dropFieldIfAllNull` parameter of the `json` method wasn't set as an option. This PR fixes that.

## How was this patch tested?

I added a test to `sql/test.py`

Author: Maxim Gekk <maxim.gekk@databricks.com>

Closes #22002 from MaxGekk/drop-field-if-all-null.
2018-08-06 16:46:55 +08:00
..
__init__.py [SPARK-22369][PYTHON][DOCS] Exposes catalog API documentation in PySpark 2017-11-02 15:22:52 +01:00
catalog.py [SPARK-23522][PYTHON] always use sys.exit over builtin exit 2018-03-08 20:38:34 +09:00
column.py [SPARK-23847][PYTHON][SQL] Add asc_nulls_first, asc_nulls_last to PySpark 2018-04-08 12:09:06 +08:00
conf.py [SPARK-24761][SQL] Adding of isModifiable() to RuntimeConfig 2018-07-11 17:38:43 -07:00
context.py [SPARK-24665][PYSPARK] Use SQLConf in PySpark to manage all sql configs 2018-07-02 14:35:37 +08:00
dataframe.py [SPARK-21274][SQL] Implement INTERSECT ALL clause 2018-07-29 22:11:01 -07:00
functions.py [SPARK-23915][SQL] Add array_except function 2018-08-02 02:52:30 +08:00
group.py [SPARK-24722][SQL] pivot() with Column type argument 2018-08-04 14:17:32 +08:00
readwriter.py [SPARK-23772][FOLLOW-UP][SQL] Provide an option to ignore column of all null values or empty array during JSON schema inference 2018-08-06 16:46:55 +08:00
session.py [SPARK-24563][PYTHON] Catch TypeError when testing existence of HiveConf when creating pysp… 2018-06-14 13:16:20 -07:00
streaming.py [SPARK-24565][SS] Add API for in Structured Streaming for exposing output rows of each microbatch as a DataFrame 2018-06-19 13:56:51 -07:00
tests.py [SPARK-23772][FOLLOW-UP][SQL] Provide an option to ignore column of all null values or empty array during JSON schema inference 2018-08-06 16:46:55 +08:00
types.py [SPARK-24057][PYTHON] put the real data type in the AssertionError message 2018-04-26 14:21:22 -07:00
udf.py [SPARK-23754][PYTHON][FOLLOWUP] Move UDF stop iteration wrapping from driver to executor 2018-06-11 10:15:42 +08:00
utils.py [SPARK-24565][SS] Add API for in Structured Streaming for exposing output rows of each microbatch as a DataFrame 2018-06-19 13:56:51 -07:00
window.py [SPARK-23861][SQL][DOC] Clarify default window frame with and without orderBy clause 2018-04-07 00:15:54 +08:00