spark-instrumented-optimizer/dev
Xinrong Meng a970f8505d [SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures
### What changes were proposed in this pull request?

The PR is proposed for **pandas APIs on Spark**, in order to separate arithmetic operations shown as below into data-type-based structures.
`__add__, __sub__, __mul__, __truediv__, __floordiv__, __pow__, __mod__,
__radd__, __rsub__, __rmul__, __rtruediv__, __rfloordiv__, __rpow__,__rmod__`

DataTypeOps and subclasses are introduced.

The existing behaviors of each arithmetic operation should be preserved.

### Why are the changes needed?

Currently, the same arithmetic operation of all data types is defined in one function, so it’s difficult to extend the behavior change based on the data types.

Introducing DataTypeOps would be the foundation for [pandas APIs on Spark: Separate basic operations into data type based structures.](https://docs.google.com/document/d/12MS6xK0hETYmrcl5b9pX5lgV4FmGVfpmcSKq--_oQlc/edit?usp=sharing).

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tests are introduced under pyspark.pandas.tests.data_type_ops. One test file per DataTypeOps class.

Closes #32596 from xinrong-databricks/datatypeop_arith_fix.

Authored-by: Xinrong Meng <xinrong.meng@databricks.com>
Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
2021-05-19 19:47:00 -07:00
..
create-release [SPARK-35425][BUILD] Pin jinja2 in spark-rm/Dockerfile and add as a required dependency in the release README.md 2021-05-18 16:48:23 +09:00
deps [SPARK-35295][ML] Replace fully com.github.fommil.netlib by dev.ludovic.netlib:2.0 2021-05-12 08:59:36 -05:00
sparktestsupport [SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures 2021-05-19 19:47:00 -07:00
tests Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
.gitignore [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*.py to .gitignore file. 2018-01-31 00:51:00 +09:00
.rat-excludes [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
.scalafmt.conf [SPARK-35255][BUILD] Automated formatting for Scala Code for Blank Lines 2021-04-30 11:45:58 +09:00
appveyor-guide.md Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
appveyor-install-dependencies.ps1 [SPARK-33105][INFRA] Change default R arch from i386 to x64 and parametrize BINPREF 2020-10-10 13:48:26 +09:00
change-scala-version.sh [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli 2021-03-18 12:31:50 +09:00
check-license [MINOR][INFRA] Suppress warning in check-license 2020-11-23 10:38:40 +09:00
checkstyle-suppressions.xml [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+ 2019-11-03 15:13:06 -08:00
checkstyle.xml [MINOR] Fix google style guide address 2019-12-12 11:04:01 -06:00
eslint.json [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
github_jira_sync.py Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
lint-java [SPARK-23063][K8S] K8s changes for publishing scripts (and a couple of other misses) 2018-01-13 21:34:28 -08:00
lint-js [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
lint-python [SPARK-35061][BUILD] Upgrade pycodestyle from 2.6.0 to 2.7.0 2021-04-14 08:07:05 -07:00
lint-r [SPARK-29932][R][TESTS] lint-r should do non-zero exit in case of errors 2019-11-17 10:09:46 -08:00
lint-r.R [MINOR][R] small tidying of sh scripts for R 2020-04-30 16:58:05 -07:00
lint-scala [SPARK-27158][BUILD] dev/mima and dev/scalastyle support dynamic profiles 2019-03-15 08:20:42 +09:00
make-distribution.sh [SPARK-31041][BUILD] Show Maven errors from within make-distribution.sh 2020-03-11 08:22:02 -05:00
merge_spark_pr.py [MINOR] Fix usage print to guide pip3 to install jira-python library 2020-09-03 01:10:59 +09:00
mima [SPARK-33510][BUILD] Update SBT to 1.4.4 2020-11-22 22:56:59 -08:00
package-lock.json [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
package.json [SPARK-35175][BUILD] Add linter for JavaScript source files 2021-05-07 21:55:08 +09:00
pip-sanity-check.py [SPARK-32319][PYSPARK] Disallow the use of unused imports 2020-08-08 08:51:57 -07:00
README.md Merge pull request #565 from pwendell/dev-scripts. Closes #565. 2014-02-08 23:13:34 -08:00
requirements.txt [MINOR][INFRA] Upgrade Jira client to 2.0.0 2021-04-18 11:34:24 +09:00
run-pip-tests [SPARK-35393][PYTHON][INFRA][TESTS] Recover pip packaging test in Github Actions 2021-05-13 10:35:56 -07:00
run-tests [SPARK-29672][PYSPARK] update spark testing framework to use python3 2019-11-14 10:18:55 -08:00
run-tests-jenkins [SPARK-33535][INFRA][TESTS] Export LANG to en_US.UTF-8 in run-tests-jenkins script 2020-11-24 09:50:10 -08:00
run-tests-jenkins.py Spelling r common dev mlib external project streaming resource managers python 2020-11-27 10:22:45 -06:00
run-tests.py [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources 2021-04-14 13:28:11 +09:00
sbt-checkstyle [SPARK-27158][BUILD] dev/mima and dev/scalastyle support dynamic profiles 2019-03-15 08:20:42 +09:00
scalafmt [SPARK-30570][BUILD] Update scalafmt plugin to 1.0.3 with onlyChangedFiles feature 2020-01-23 12:44:43 -08:00
scalastyle Revert "[SPARK-30534][INFRA] Use mvn in dev/scalastyle" 2020-01-21 18:23:03 +09:00
test-dependencies.sh [SPARK-20202][BUILD][SQL] Remove references to org.spark-project.hive (Hive 1.2.1) 2020-10-05 15:29:56 -07:00
tox.ini [SPARK-33749][BUILD][PYTHON] Exclude target directory in pycodestyle and flake8 2020-12-11 14:15:56 +09:00

Spark Developer Scripts

This directory contains scripts useful to developers when packaging, testing, or committing to Spark.

Many of these scripts require Apache credentials to work correctly.