[SPARK-24624][SQL][PYTHON] Support mixture of Python UDF and Scalar Pandas UDF
## What changes were proposed in this pull request?
This PR add supports for using mixed Python UDF and Scalar Pandas UDF, in the following two cases:
(1)
```
from pyspark.sql.functions import udf, pandas_udf
udf('int')
def f1(x):
return x + 1
pandas_udf('int')
def f2(x):
return x + 1
df = spark.range(0, 1).toDF('v') \
.withColumn('foo', f1(col('v'))) \
.withColumn('bar', f2(col('v')))
```
QueryPlan:
```
>>> df.explain(True)
== Parsed Logical Plan ==
'Project [v#2L, foo#5, f2('v) AS bar#9]
+- AnalysisBarrier
+- Project [v#2L, f1(v#2L) AS foo#5]
+- Project [id#0L AS v#2L]
+- Range (0, 1, step=1, splits=Some(4))
== Analyzed Logical Plan ==
v: bigint, foo: int, bar: int
Project [v#2L, foo#5, f2(v#2L) AS bar#9]
+- Project [v#2L, f1(v#2L) AS foo#5]
+- Project [id#0L AS v#2L]
+- Range (0, 1, step=1, splits=Some(4))
== Optimized Logical Plan ==
Project [id#0L AS v#2L, f1(id#0L) AS foo#5, f2(id#0L) AS bar#9]
+- Range (0, 1, step=1, splits=Some(4))
== Physical Plan ==
*(2) Project [id#0L AS v#2L, pythonUDF0#13 AS foo#5, pythonUDF0#14 AS bar#9]
+- ArrowEvalPython [f2(id#0L)], [id#0L, pythonUDF0#13, pythonUDF0#14]
+- BatchEvalPython [f1(id#0L)], [id#0L, pythonUDF0#13]
+- *(1) Range (0, 1, step=1, splits=4)
```
(2)
```
from pyspark.sql.functions import udf, pandas_udf
udf('int')
def f1(x):
return x + 1
pandas_udf('int')
def f2(x):
return x + 1
df = spark.range(0, 1).toDF('v')
df = df.withColumn('foo', f2(f1(df['v'])))
```
QueryPlan:
```
>>> df.explain(True)
== Parsed Logical Plan ==
Project [v#21L, f2(f1(v#21L)) AS foo#46]
+- AnalysisBarrier
+- Project [v#21L, f1(f2(v#21L)) AS foo#39]
+- Project [v#21L, <lambda>(<lambda>(v#21L)) AS foo#32]
+- Project [v#21L, <lambda>(<lambda>(v#21L)) AS foo#25]
+- Project [id#19L AS v#21L]
+- Range (0, 1, step=1, splits=Some(4))
== Analyzed Logical Plan ==
v: bigint, foo: int
Project [v#21L, f2(f1(v#21L)) AS foo#46]
+- Project [v#21L, f1(f2(v#21L)) AS foo#39]
+- Project [v#21L, <lambda>(<lambda>(v#21L)) AS foo#32]
+- Project [v#21L, <lambda>(<lambda>(v#21L)) AS foo#25]
+- Project [id#19L AS v#21L]
+- Range (0, 1, step=1, splits=Some(4))
== Optimized Logical Plan ==
Project [id#19L AS v#21L, f2(f1(id#19L)) AS foo#46]
+- Range (0, 1, step=1, splits=Some(4))
== Physical Plan ==
*(2) Project [id#19L AS v#21L, pythonUDF0#50 AS foo#46]
+- ArrowEvalPython [f2(pythonUDF0#49)], [id#19L, pythonUDF0#49, pythonUDF0#50]
+- BatchEvalPython [f1(id#19L)], [id#19L, pythonUDF0#49]
+- *(1) Range (0, 1, step=1, splits=4)
```
## How was this patch tested?
New tests are added to BatchEvalPythonExecSuite and ScalarPandasUDFTests
Author: Li Jin <ice.xelloss@gmail.com>
Closes #21650 from icexelloss/SPARK-24624-mix-udf.
e8752095a0