ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-36104][PYTHON] Manage InternalField in DataTypeOps.neg/abs

Browse source 
### What changes were proposed in this pull request?
Manage InternalField for DataTypeOps.neg/abs.

### Why are the changes needed?
The spark data type and nullability must be the same as the original when DataTypeOps.neg/abs.
We should manage InternalField for this case.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Unit tests.

Closes #33307 from xinrong-databricks/internalField.

Authored-by: Xinrong Meng <xinrong.meng@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This commit is contained in:
Xinrong Meng 2021-07-13 12:07:05 +09:00 committed by Hyukjin Kwon
parent 8ab1933fd7
commit 5afc27f899
1 changed files with 4 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
python/pyspark/pandas/data_type_ops/num_ops.py
View file

@ -123,10 +123,12 @@ class NumericOps(DataTypeOps):
return column_op(rmod)(left, right)
def neg(self, operand: IndexOpsLike) -> IndexOpsLike:
return cast(IndexOpsLike, column_op(Column.__neg__)(operand))
return operand._with_new_scol(-operand.spark.column, field=operand._internal.data_fields[0])
def abs(self, operand: IndexOpsLike) -> IndexOpsLike:
return cast(IndexOpsLike, column_op(F.abs)(operand))
return operand._with_new_scol(
F.abs(operand.spark.column), field=operand._internal.data_fields[0]
)
def lt(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
return column_op(Column.__lt__)(left, right)