ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
spark-instrumented-optimizer/python/pyspark/pandas/tests
History
Xinrong Meng af81ad0d7e [SPARK-36001][PYTHON] Assume result's index to be disordered in tests with operations on different Series 
### What changes were proposed in this pull request?
For tests with operations on different Series, sort index of results before comparing them with pandas.

### Why are the changes needed?
We have many tests with operations on different Series in `spark/python/pyspark/pandas/tests/data_type_ops/` that assume the result's index to be sorted and then compare to the pandas' behavior.

The assumption on the result's index ordering is wrong since Spark DataFrame join is used internally and the order is not preserved if the data being in different partitions.

So we should assume the result to be disordered and sort the index of such results before comparing them with pandas.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Unit tests.

Closes #33274 from xinrong-databricks/datatypeops_testdiffframe.

Authored-by: Xinrong Meng <xinrong.meng@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
2021-07-09 12:42:48 +09:00
..
data_type_ops [SPARK-36001][PYTHON] Assume result's index to be disordered in tests with operations on different Series 2021-07-09 12:42:48 +09:00
indexes [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations 2021-07-08 12:27:48 -07:00
plot [SPARK-35738][PYTHON] Support 'y' properly in DataFrame with non-numeric columns with plots 2021-06-12 14:36:46 +09:00
__init__.py [SPARK-34886][PYTHON] Port/integrate Koalas DataFrame unit test into PySpark 2021-04-09 15:48:13 +09:00
test_categorical.py [SPARK-35499][PYTHON] Apply black to pandas API on Spark codes 2021-06-06 17:30:07 -07:00
test_config.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_csv.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_dataframe.py [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations 2021-07-08 12:27:48 -07:00
test_dataframe_conversion.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_dataframe_spark_io.py [SPARK-35499][PYTHON] Apply black to pandas API on Spark codes 2021-06-06 17:30:07 -07:00
test_default_index.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_expanding.py [SPARK-35499][PYTHON] Apply black to pandas API on Spark codes 2021-06-06 17:30:07 -07:00
test_extension.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_frame_spark.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_groupby.py Revert "[SPARK-35721][PYTHON] Path level discover for python unittests" 2021-06-29 12:08:09 -07:00
test_indexing.py Revert "[SPARK-35721][PYTHON] Path level discover for python unittests" 2021-06-29 12:08:09 -07:00
test_indexops_spark.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_internal.py [SPARK-35343][PYTHON] Make the conversion from/to pandas data-type-based for non-ExtensionDtypes 2021-06-07 13:12:12 -07:00
test_namespace.py [SPARK-35812][PYTHON] Throw ValueError if version and timestamp are used together in to_delta 2021-06-23 19:04:45 +09:00
test_numpy_compat.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_ops_on_diff_frames.py Revert "[SPARK-35721][PYTHON] Path level discover for python unittests" 2021-06-29 12:08:09 -07:00
test_ops_on_diff_frames_groupby.py Revert "[SPARK-35721][PYTHON] Path level discover for python unittests" 2021-06-29 12:08:09 -07:00
test_ops_on_diff_frames_groupby_expanding.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_ops_on_diff_frames_groupby_rolling.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_repr.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_reshape.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_rolling.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_series.py [SPARK-35339][PYTHON] Improve unit tests for data-type-based basic operations 2021-07-01 17:37:32 -07:00
test_series_conversion.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_series_datetime.py [SPARK-35340][PYTHON] Standardize TypeError messages for unsupported basic operations 2021-07-08 12:27:48 -07:00
test_series_string.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_spark_functions.py [SPARK-35344][PYTHON] Support creating a Column of numpy literals in pandas API on Spark 2021-06-28 19:03:42 -07:00
test_sql.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_stats.py [SPARK-35981][PYTHON][TEST] Use check_exact=False to loosen the check precision 2021-07-02 17:58:10 +09:00
test_typedef.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_utils.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00
test_window.py [SPARK-35364][PYTHON] Renaming the existing Koalas related codes 2021-05-20 15:08:30 -07:00