6e4e04f2a1
### What changes were proposed in this pull request? Make unary and comparison operators data-type-based. Refactored operators include: - Unary operators: `__neg__`, `__abs__`, `__invert__`, - Comparison operators: `>`, `>=`, `<`, `<=`, `==`, `!=` Non-goal: Tasks below are inspired during the development of this PR. [[SPARK-35997] Implement comparison operators for CategoricalDtype in pandas API on Spark](https://issues.apache.org/jira/browse/SPARK-35997) [[SPARK-36000] Support creating a ps.Series/Index with `Decimal('NaN')` with Arrow disabled](https://issues.apache.org/jira/browse/SPARK-36000) [[SPARK-36001] Assume result's index to be disordered in tests with operations on different Series](https://issues.apache.org/jira/browse/SPARK-36001) [[SPARK-36002] Consolidate tests for data-type-based operations of decimal Series](https://issues.apache.org/jira/browse/SPARK-36002) [[SPARK-36003] Implement unary operator `invert` of numeric ps.Series/Index](https://issues.apache.org/jira/browse/SPARK-36003) ### Why are the changes needed? We have been refactoring basic operators to be data-type-based for readability, flexibility, and extensibility. Unary and comparison operators are still not data-type-based yet. We should fill the gaps. ### Does this PR introduce _any_ user-facing change? Yes. - Better error messages. For example, Before: ```py >>> import pyspark.pandas as ps >>> psser = ps.Series([b"2", b"3", b"4"]) >>> -psser Traceback (most recent call last): ... pyspark.sql.utils.AnalysisException: cannot resolve '(- `0`)' due to data type mismatch: ... ``` After: ```py >>> import pyspark.pandas as ps >>> psser = ps.Series([b"2", b"3", b"4"]) >>> -psser Traceback (most recent call last): ... TypeError: Unary - can not be applied to binaries. >>> ``` - Support unary `-` of `bool` Series. For example, Before: ```py >>> psser = ps.Series([True, False, True]) >>> -psser Traceback (most recent call last): ... pyspark.sql.utils.AnalysisException: cannot resolve '(- `0`)' due to data type mismatch: ... ``` After: ```py >>> psser = ps.Series([True, False, True]) >>> -psser 0 False 1 True 2 False dtype: bool ``` ### How was this patch tested? Unit tests. Closes #33162 from xinrong-databricks/datatypeops_refactor. Authored-by: Xinrong Meng <xinrong.meng@databricks.com> Signed-off-by: Takuya UESHIN <ueshin@databricks.com> |
||
---|---|---|
.. | ||
data_type_ops | ||
indexes | ||
missing | ||
plot | ||
spark | ||
tests | ||
typedef | ||
usage_logging | ||
__init__.py | ||
_typing.py | ||
accessors.py | ||
base.py | ||
categorical.py | ||
config.py | ||
datetimes.py | ||
exceptions.py | ||
extensions.py | ||
frame.py | ||
generic.py | ||
groupby.py | ||
indexing.py | ||
internal.py | ||
ml.py | ||
mlflow.py | ||
namespace.py | ||
numpy_compat.py | ||
series.py | ||
sql_processor.py | ||
strings.py | ||
utils.py | ||
window.py |