### What changes were proposed in this pull request?
Add new built-in SQL functions: secant and cosecant, and add them as Scala and Python functions.
### Why are the changes needed?
Cotangent has been supported in Spark SQL but Secant and Cosecant are missing though I believe they can be used as much as cot.
Related Links: [SPARK-20751](https://github.com/apache/spark/pull/17999) [SPARK-36660](https://github.com/apache/spark/pull/33906)
### Does this PR introduce _any_ user-facing change?
Yes, users can now use these functions.
### How was this patch tested?
Unit tests
Closes#33988 from yutoacts/SPARK-36683.
Authored-by: Yuto Akutsu <yuto.akutsu@oss.nttdata.com>
Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>
### What changes were proposed in this pull request?
This PR group exception messages in core/src/main/scala/org/apache/spark/api
### Why are the changes needed?
It will largely help with standardization of error messages and its maintenance.
### Does this PR introduce _any_ user-facing change?
No. Error messages remain unchanged.
### How was this patch tested?
No new tests - pass all original tests to make sure it doesn't break any existing behavior.
Closes#33536 from dgd-contributor/SPARK-36101.
Authored-by: dgd-contributor <dgd_contributor@viettel.com.vn>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
For query
```
select array_intersect(array(cast('nan' as double), 1d), array(cast('nan' as double)))
```
This returns [NaN], but it should return [].
This issue is caused by `OpenHashSet` can't handle `Double.NaN` and `Float.NaN` too.
In this pr fix this based on https://github.com/apache/spark/pull/33955
### Why are the changes needed?
Fix bug
### Does this PR introduce _any_ user-facing change?
ArrayIntersect won't show equal `NaN` value
### How was this patch tested?
Added UT
Closes#33995 from AngersZhuuuu/SPARK-36754.
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
This PR aims to upgrade Apache ORC from 1.6.11 to 1.7.0 for Apache Spark 3.3.0.
### Why are the changes needed?
[Apache ORC 1.7.0](https://orc.apache.org/news/2021/09/15/ORC-1.7.0/) is a new release with the following new features and improvements.
- ORC-377 Support Snappy compression in C++ Writer
- ORC-577 Support row-level filtering
- ORC-716 Build and test on Java 17-EA
- ORC-731 Improve Java Tools
- ORC-742 LazyIO of non-filter columns
- ORC-751 Implement Predicate Pushdown in C++ Reader
- ORC-755 Introduce OrcFilterContext
- ORC-757 Add Hashtable implementation for dictionary
- ORC-780 Support LZ4 Compression in C++ Writer
- ORC-797 Allow writers to get the stripe information
- ORC-818 Build and test in Apple Silicon
- ORC-861 Bump CMake minimum requirement to 2.8.12
- ORC-867 Upgrade hive-storage-api to 2.8.1
- ORC-984 Save the software version that wrote each ORC file
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the existing CIs because this is a dependency change.
Closes#34045 from dongjoon-hyun/SPARK-34112.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request?
This PR proposes the new syntax introduced in https://github.com/apache/spark/pull/33954. Namely, users now can specify the index type and name as below:
```python
import pandas as pd
import pyspark.pandas as ps
def transform(pdf) -> pd.DataFrame[int, [int, int]]:
pdf['A'] = pdf.id + 1
return pdf
ps.range(5).koalas.apply_batch(transform)
```
```
c0 c1
0 0 1
1 1 2
2 2 3
3 3 4
4 4 5
```
```python
import pandas as pd
import pyspark.pandas as ps
def transform(pdf) -> pd.DataFrame[("index", int), [("a", int), ("b", int)]]:
pdf['A'] = pdf.id * pdf.id
return pdf
ps.range(5).koalas.apply_batch(transform)
```
```
a b
index
0 0 0
1 1 1
2 2 4
3 3 9
4 4 16
```
Again, this syntax remains experimental and this is a non-standard way apart from Python standard. We should migrate to proper typing once pandas supports it like `numpy.typing`.
### Why are the changes needed?
The rationale is described in https://github.com/apache/spark/pull/33954. In order to avoid unnecessary computation for default index or schema inference.
### Does this PR introduce _any_ user-facing change?
Yes, this PR affects the following APIs:
- `DataFrame.apply(..., axis=1)`
- `DataFrame.groupby.apply(...)`
- `DataFrame.pandas_on_spark.transform_batch(...)`
- `DataFrame.pandas_on_spark.apply_batch(...)`
Now they can specify the index type with the new syntax below:
```
DataFrame[index_type, [type, ...]]
DataFrame[(index_name, index_type), [(name, type), ...]]
DataFrame[dtype instance, dtypes instance]
DataFrame[(index_name, index_type), zip(names, types)]
```
### How was this patch tested?
Manually tested, and unittests were added.
Closes#34007 from HyukjinKwon/SPARK-36710.
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
Refactor some exceptions in QueryExecutionErrors to use error classes. as follows:
```
columnChangeUnsupportedError
logicalHintOperatorNotRemovedDuringAnalysisError
cannotEvaluateExpressionError
cannotGenerateCodeForExpressionError
cannotTerminateGeneratorError
castingCauseOverflowError
cannotChangeDecimalPrecisionError
invalidInputSyntaxForNumericError
cannotCastFromNullTypeError
cannotCastError
cannotParseDecimalError
simpleStringWithNodeIdUnsupportedError
evaluateUnevaluableAggregateUnsupportedError
dataTypeUnsupportedError
dataTypeUnsupportedError
failedExecuteUserDefinedFunctionError
divideByZeroError
invalidArrayIndexError
mapKeyNotExistError
rowFromCSVParserNotExpectedError
```
### Why are the changes needed?
[SPARK-36107](https://issues.apache.org/jira/browse/SPARK-36107)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existed UT Testcase
Closes#33538 from Peng-Lei/SPARK-36017.
Lead-authored-by: PengLei <peng.8lei@gmail.com>
Co-authored-by: Lei Peng <peng.8lei@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
Remove the appAttemptId from TransportConf, and parsing through SparkEnv.
### Why are the changes needed?
Push based shuffle will fail if there are any attemptId set in the SparkConf, as the attemptId is not set correctly in Driver.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tested within our Yarn cluster. Without this PR, the Driver will fail to finalize the shuffle merge on all the mergers. After the patch, Driver can successfully finalize the shuffle merge and the push based shuffle can work fine.
Also with unit test to verify the attemptId is being set in the BlockStoreClient in Driver.
Closes#34018 from zhouyejoe/SPARK-36772.
Authored-by: Ye Zhou <yezhou@linkedin.com>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
### What changes were proposed in this pull request?
As a followup of #34025 to remove duplicate test.
### Why are the changes needed?
To remove duplicate test.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing test.
Closes#34032 from viirya/remove.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
### What changes were proposed in this pull request?
Java 17 has been officially released. This PR makes `dev/mima` runs on Java 17.
### Why are the changes needed?
To make tests pass on Java 17.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test.
Closes#34022 from RabbidHY/SPARK-36780.
Lead-authored-by: Yang He <stitch106hy@gmail.com>
Co-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request?
This PR aims to support number-only column names in ORC data sources.
In the current master, with ORC datasource, we can write a DataFrame which contains such columns into ORC files.
```
spark.sql("SELECT 'a' as `1`").write.orc("/path/to/dest")
```
But reading the ORC files will fail.
```
spark.read.orc("/path/to/dest")
...
== SQL ==
struct<1:string>
-------^^^
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:265)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:126)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:40)
at org.apache.spark.sql.execution.datasources.orc.OrcUtils$.org$apache$spark$sql$execution$datasources$orc$OrcUtils$$toCatalystSchema(OrcUtils.scala:91)
```
The cause of this is `OrcUtils.toCatalystSchema` fails to handle such column names.
In `OrcUtils.toCatalystSchema`, `CatalystSqlParser` is used to create a instance of `StructType` which represents a schema but `CatalystSqlParser.parseDataType` fails to parse if a column name (and nested field) consists of only numbers.
### Why are the changes needed?
For better usability.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests.
Closes#33915 from sarutak/fix-orc-schema-issue.
Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
Add comment about how ArrayMin/ArrayMax/SortArray/ArraySort handle NaN and add Unit test for this
### Why are the changes needed?
Add Unit test
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added UT
Closes#34008 from AngersZhuuuu/SPARK-36740.
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
This patch proposes to fix incorrect schema of `union`.
### Why are the changes needed?
The current `union` result of nested struct columns is incorrect. By definition of `union` API, it should resolve columns by position, not by name. Right now when determining the `output` (aka. the schema) of union plan, we use `merge` API which actually merges two structs (simply think it as concatenate fields from two structs if not overlapping). The merging behavior doesn't match the `union` definition.
So currently we get incorrect schema but the query result is correct. We should fix the incorrect schema.
### Does this PR introduce _any_ user-facing change?
Yes, fixing a bug of incorrect schema.
### How was this patch tested?
Added unit test.
Closes#34025 from viirya/SPARK-36673.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
The `CollapseProject` rule can combine adjacent projects and merge the project lists. The key idea behind this rule is that the evaluation of project is relatively expensive, and that expression evaluation is cheap and that the expression duplication caused by this rule is not a problem. This last assumption is, unfortunately, not always true:
- A user can invoke some expensive UDF, this now gets invoked more often than originally intended.
- A projection is very cheap in whole stage code generation. The duplication caused by `CollapseProject` does more harm than good here.
This PR addresses this problem, by only collapsing projects when it does not duplicate expensive expressions. In practice this means an input reference may only be consumed once, or when its evaluation does not incur significant overhead (currently attributes, nested column access, aliases & literals fall in this category).
### Why are the changes needed?
We have seen multiple complains about `CollapseProject` in the past, due to it may duplicate expensive expressions. The most recent one is https://github.com/apache/spark/pull/33903 .
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
a new UT and existing test
Closes#33958 from cloud-fan/collapse.
Authored-by: Wenchen Fan <wenchen@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
The test “ensure continuous stream is being used“ in KafkaContinuousTest quickly checks the actual type of the execution, and stops the query. Stopping the streaming query in continuous mode is done by interrupting query execution thread and join with indefinite timeout.
In parallel, started streaming query is going to generate execution plan, including running optimizer. Some parts of SessionState can be built at that time, as they are defined as lazy. The problem is, some of them seem to “swallow” the InterruptedException and let the thread run continuously.
That said, the query can’t indicate whether there is a request on stopping query, so the query won’t stop.
This PR fixes such scenario via ensuring that streaming query has started before the test stops the query.
### Why are the changes needed?
Race-condition could end up with test hang till test framework marks it as timed-out.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes#34004 from HeartSaVioR/SPARK-36764.
Authored-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
For query
```
select array_distinct(array(cast('nan' as double), cast('nan' as double)))
```
This returns [NaN, NaN], but it should return [NaN].
This issue is caused by `OpenHashSet` can't handle `Double.NaN` and `Float.NaN` too.
In this pr fix this based on https://github.com/apache/spark/pull/33955
### Why are the changes needed?
Fix bug
### Does this PR introduce _any_ user-facing change?
ArrayDistinct won't show duplicated `NaN` value
### How was this patch tested?
Added UT
Closes#33993 from AngersZhuuuu/SPARK-36741.
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
Support ILIKE (case insensitive LIKE) API on Scala.
### Why are the changes needed?
ILIKE statement on SQL interface is supported by SPARK-36674.
This PR will support Scala(dataframe) API for it.
### Does this PR introduce _any_ user-facing change?
Yes. Users can call `ilike` from dataframe.
### How was this patch tested?
unit tests.
Closes#34027 from yoda-mon/scala-ilike.
Authored-by: Leona Yoda <yodal@oss.nttdata.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
### What changes were proposed in this pull request?
In array functions, we use constant 0 as the placeholder when adding a null value to an array buffer. This PR makes sure the constant 0 matches the type of the array element.
### Why are the changes needed?
Fix a potential bug. Somehow we can hit this bug sometimes after https://github.com/apache/spark/pull/33955 .
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
existing tests
Closes#34029 from cloud-fan/minor.
Authored-by: Wenchen Fan <wenchen@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
This is a re-work of https://github.com/apache/spark/pull/30003, here we add support for writing Hive bucketed table with Parquet/ORC file format (data source v1 write path and Hive hash as the hash function). Support for Hive's other file format will be added in follow up PR.
The changes are mostly on:
* `HiveMetastoreCatalog.scala`: When converting hive table relation to data source relation, pass bucket info (BucketSpec) and other hive related info as options into `HadoopFsRelation` and `LogicalRelation`, which can be later accessed by `FileFormatWriter` to customize bucket id and file name.
* `FileFormatWriter.scala`: Use `HiveHash` for `bucketIdExpression` if it's writing to Hive bucketed table. In addition, Spark output file name should follow Hive/Presto/Trino bucketed file naming convention. Introduce another parameter `bucketFileNamePrefix` and it introduces subsequent change in `FileFormatDataWriter`.
* `HadoopMapReduceCommitProtocol`: Implement the new file name APIs introduced in https://github.com/apache/spark/pull/33012, and change its sub-class `PathOutputCommitProtocol`, to make Hive bucketed table writing work with all commit protocol (including S3A commit protocol).
### Why are the changes needed?
To make Spark write other-SQL-engines-compatible bucketed table. Currently Spark bucketed table cannot be leveraged by other SQL engines like Hive and Presto, because it uses a different hash function (Spark murmur3hash) and different file name scheme. With this PR, the Spark-written-Hive-bucketed-table can be efficiently read by Presto and Hive to do bucket filter pruning, join, group-by, etc. This was and is blocking several companies (confirmed from Facebook, Lyft, etc) migrate bucketing workload from Hive to Spark.
### Does this PR introduce _any_ user-facing change?
Yes, any Hive bucketed table (with Parquet/ORC format) written by Spark, is properly bucketed and can be efficiently processed by Hive and Presto/Trino.
### How was this patch tested?
* Added unit test in BucketedWriteWithHiveSupportSuite.scala, to verify bucket file names and each row in each bucket is written properly.
* Tested by Lyft Spark team (Shashank Pedamallu) to read Spark-written bucketed table from Trino, Spark and Hive.
Closes#33432 from c21/hive-bucket-v1.
Authored-by: Cheng Su <chengsu@fb.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
This PR suppresses the warnings for plans where AQE is not supported. Currently we show the warnings such as:
```
org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324881 DESC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324881]
```
for every plan that AQE is not supported.
### Why are the changes needed?
It's too noisy now. Below is the example of `SortSuite` run:
```
14:51:40.675 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324881 DESC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324881]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=true, sortOrder=List('a DESC NULLS FIRST) (785 milliseconds)
14:51:41.416 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: ReferenceSort [a#324884 ASC NULLS FIRST], true
+- Scan ExistingRDD[a#324884]
.
14:51:41.467 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324884 ASC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324884]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=false, sortOrder=List('a ASC NULLS FIRST) (796 milliseconds)
14:51:42.210 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: ReferenceSort [a#324887 ASC NULLS LAST], true
+- Scan ExistingRDD[a#324887]
.
14:51:42.259 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324887 ASC NULLS LAST], true, 23
+- Scan ExistingRDD[a#324887]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=false, sortOrder=List('a ASC NULLS LAST) (797 milliseconds)
14:51:43.009 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: ReferenceSort [a#324890 DESC NULLS LAST], true
+- Scan ExistingRDD[a#324890]
.
14:51:43.061 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324890 DESC NULLS LAST], true, 23
+- Scan ExistingRDD[a#324890]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=false, sortOrder=List('a DESC NULLS LAST) (848 milliseconds)
14:51:43.857 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: ReferenceSort [a#324893 DESC NULLS FIRST], true
+- Scan ExistingRDD[a#324893]
.
14:51:43.903 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324893 DESC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324893]
.
[info] - sorting on DayTimeIntervalType(0,1) with nullable=false, sortOrder=List('a DESC NULLS FIRST) (827 milliseconds)
14:51:44.682 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: ReferenceSort [a#324896 ASC NULLS FIRST], true
+- Scan ExistingRDD[a#324896]
.
14:51:44.748 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324896 ASC NULLS FIRST], true, 23
+- Scan ExistingRDD[a#324896]
.
[info] - sorting on YearMonthIntervalType(0,1) with nullable=true, sortOrder=List('a ASC NULLS FIRST) (565 milliseconds)
14:51:45.248 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: ReferenceSort [a#324899 ASC NULLS LAST], true
+- Scan ExistingRDD[a#324899]
.
14:51:45.312 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324899 ASC NULLS LAST], true, 23
+- Scan ExistingRDD[a#324899]
.
[info] - sorting on YearMonthIntervalType(0,1) with nullable=true, sortOrder=List('a ASC NULLS LAST) (591 milliseconds)
14:51:45.841 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: ReferenceSort [a#324902 DESC NULLS LAST], true
+- Scan ExistingRDD[a#324902]
.
14:51:45.905 WARN org.apache.spark.sql.execution.adaptive.InsertAdaptiveSparkPlan: spark.sql.adaptive.enabled is enabled but is not supported for query: Sort [a#324902 DESC NULLS LAST], true, 23
+- Scan ExistingRDD[a#324902]
.
```
### Does this PR introduce _any_ user-facing change?
Yes, it will show less warnings to users. Note that AQE is enabled by default from Spark 3.2, see SPARK-33679
### How was this patch tested?
Manually tested via unittests.
Closes#34026 from HyukjinKwon/minor-log-level.
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
`ScanOperation` collects adjacent Projects and Filters. The caller side always assume that the collected Filters should run before collected Projects, which means `ScanOperation` effectively pushes Filter through Project.
Following `PushPredicateThroughNonJoin`, we should not push Filter through nondeterministic Project. This PR fixes `ScanOperation` to follow this rule.
### Why are the changes needed?
Fix a bug that violates the semantic of nondeterministic expressions.
### Does this PR introduce _any_ user-facing change?
Most likely no change, but in some cases, this is a correctness bug fix which changes the query result.
### How was this patch tested?
existing tests
Closes#34023 from cloud-fan/scan.
Authored-by: Wenchen Fan <wenchen@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
Change the unit test for parquet compression
### Why are the changes needed?
To check the compression for parquet
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
change unit test
Closes#34012 from BelodengKlaus/spark36773.
Authored-by: BelodengKlaus <jp.xiong520@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
Fix when list of data type tuples has len = 1
### Why are the changes needed?
Fix when list of data type tuples has len = 1
``` python
>>> ps.DataFrame[("a", int), [int]]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, int]
>>> ps.DataFrame[("a", int), [("b", int)]]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/dgd/spark/python/pyspark/pandas/frame.py", line 11998, in __class_getitem__
return create_tuple_for_frame_type(params)
File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line 685, in create_tuple_for_frame_type
return Tuple[extract_types(params)]
File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line 755, in extract_types
return (index_type,) + extract_types(data_types)
File "/Users/dgd/spark/python/pyspark/pandas/typedef/typehints.py", line 770, in extract_types
raise TypeError(
TypeError: Type hints should be specified as one of:
- DataFrame[type, type, ...]
- DataFrame[name: type, name: type, ...]
- DataFrame[dtypes instance]
- DataFrame[zip(names, types)]
- DataFrame[index_type, [type, ...]]
- DataFrame[(index_name, index_type), [(name, type), ...]]
- DataFrame[dtype instance, dtypes instance]
- DataFrame[(index_name, index_type), zip(names, types)]
However, got [('b', <class 'int'>)].
```
### Does this PR introduce _any_ user-facing change?
After:
``` python
>>> ps.DataFrame[("a", int), [("b", int)]]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, pyspark.pandas.typedef.typehints.NameType]
```
### How was this patch tested?
exist test
Closes#34019 from dgd-contributor/fix_when_list_of_tuple_data_type_have_len=1.
Authored-by: dgd-contributor <dgd_contributor@viettel.com.vn>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
This PR refactors test code in order to improve the debugability of `SparkSubmitSuite`.
The `sql/hive` module contains a `SparkSubmitTestUtils` helper class which launches `spark-submit` and captures its output in order to display better error messages when tests fail. This helper is currently used by `HiveSparkSubmitSuite` and `HiveExternalCatalogVersionsSuite`, but isn't used by `SparkSubmitSuite`.
In this PR, I moved `SparkSubmitTestUtils` and `ProcessTestUtils` into the `core` module and updated `SparkSubmitSuite`, `BufferHolderSparkSubmitSuite`, and `WholestageCodegenSparkSubmitSuite` to use the relocated helper classes. This required me to change `SparkSubmitTestUtils` to make its timeouts configurable and to generalize its method for locating the `spark-submit` binary.
### Why are the changes needed?
Previously, `SparkSubmitSuite` tests would fail with messages like:
```
[info] - launch simple application with spark-submit *** FAILED *** (1 second, 832 milliseconds)
[info] Process returned with exit code 101. See the log4j logs for more detail. (SparkSubmitSuite.scala:1551)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
```
which require the Spark developer to hunt in log4j logs in order to view the logs from the failed `spark-submit` command.
After this change, those tests will fail with detailed error messages that include the text of failed command plus timestamped logs captured from the failed proces:
```
[info] - launch simple application with spark-submit *** FAILED *** (2 seconds, 800 milliseconds)
[info] spark-submit returned with exit code 101.
[info] Command line: '/Users/joshrosen/oss-spark/bin/spark-submit' '--class' 'invalidClassName' '--name' 'testApp' '--master' 'local' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-0a8a0c93-3aaf-435d-9cf3-b97abd318d91/testJar-1631768004882.jar'
[info]
[info] 2021-09-15 21:53:26.041 - stderr> SLF4J: Class path contains multiple SLF4J bindings.
[info] 2021-09-15 21:53:26.042 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/oss-spark/assembly/target/scala-2.12/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[info] 2021-09-15 21:53:26.042 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[info] 2021-09-15 21:53:26.042 - stderr> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
[info] 2021-09-15 21:53:26.042 - stderr> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[info] 2021-09-15 21:53:26.619 - stderr> Error: Failed to load class invalidClassName. (SparkSubmitTestUtils.scala:97)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
I manually ran the affected test suites.
Closes#34013 from JoshRosen/SPARK-36774-move-SparkSubmitTestUtils-to-core.
Authored-by: Josh Rosen <joshrosen@databricks.com>
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
### What changes were proposed in this pull request?
As a follow up of #33975, this fixes a few indentation in DynamicPartitionPruningSuite.
### Why are the changes needed?
Fix wrong indentation.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests.
Closes#34016 from viirya/fix-style.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request?
This is a follow-up to fix the leftover during switching the Scala version.
### Why are the changes needed?
This should be consistent.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This is not tested by UT. We need to check manually. There is no more `2.12.14`.
```
$ git grep 2.12.14
R/pkg/tests/fulltests/test_sparkSQL.R: c(as.Date("2012-12-14"), as.Date("2013-12-15"), as.Date("2014-12-16")))
data/mllib/ridge-data/lpsa.data:3.5307626,0.987291634724086 -0.36279314978779 -0.922212414640967 0.232904453212813 -0.522940888712441 1.79270085261407 0.342627053981254 1.26288870310799
sql/hive/src/test/resources/data/files/over10k:-3|454|65705|4294967468|62.12|14.32|true|mike white|2013-03-01 09:11:58.703087|40.18|joggying
```
Closes#34020 from dongjoon-hyun/SPARK-36759-2.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request?
remove `CreateNamespaceStatement`
### Why are the changes needed?
remove unused code
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
existing tests
Closes#34015 from huaxingao/rm_create_ns_stmt.
Authored-by: Huaxin Gao <huaxin_gao@apple.com>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
### What changes were proposed in this pull request?
This PR aims to move Java 17 on GA from early access release to LTS release.
### Why are the changes needed?
Java 17 LTS was released a few days ago and it's available on GA.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
GA itself.
Closes#34017 from sarutak/ga-java17.
Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Yuming Wang <yumwang@ebay.com>
### What changes were proposed in this pull request?
This fixes the info log message output when starting a WebUI server
### Why are the changes needed?
This is needed by the user to go to the right location of the started service
```
21/08/05 14:33:30 INFO HistoryServer: Bound HistoryServer to 0.0.0.0, and started at http://tgudivad-mn1.test.biz:18080
```
### Does this PR introduce _any_ user-facing change?
Yes, fixes the URL displayed in the logs when starting the service.
### How was this patch tested?
Tested by running an instance of HistoryServer
Closes#33659 from thejdeep/SPARK-36433.
Authored-by: Thejdeep Gudivada <tgudivada@linkedin.com>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
### What changes were proposed in this pull request?
Add documentation for ANSI store assignment rules for
- the valid source/target type combinations
- runtime error will happen on numberic overflow
### Why are the changes needed?
Better docs
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Build docs and preview:
Closes#34014 from gengliangwang/addStoreAssignDoc.
Authored-by: Gengliang Wang <gengliang@apache.org>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
This PR aims to upgrade Apache ORC to 1.6.11 to bring the latest bug fixes.
### Why are the changes needed?
Apache ORC 1.6.11 has the following fixes.
- https://issues.apache.org/jira/projects/ORC/versions/12350499
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes#33971 from dongjoon-hyun/SPARK-36732.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request?
This PR updates `ExtractEquiJoinKeys` to return an extra field for the join condition with join keys.
### Why are the changes needed?
Sometimes we need to restore the original join condition. Before this PR, we need to build `EqualTo` expressions with the join keys, which is not always the original join condition. E.g. `EqualNullSafe(a, b)` will become `EqualTo(Coalesce(a, lit), Coalesce(b, lit))`. After this PR, we can simply use the new returned field.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests.
Closes#33985 from YannisSismanis/SPARK-36475-fix.
Authored-by: Yannis Sismanis <yannis.sismanis@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
This patch proposes to adjust the current overhead of cached relation for DPP.
### Why are the changes needed?
Currently we calculate if there is benefit of pruning with DPP by simply summing up the size of all scan relations as the overhead. However, for cached relations, the overhead should be different than a non-cached relation. This proposes to use adjusted overhead for cached relation with DPP.
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
Added unit test.
Closes#33975 from viirya/reduce-cache-overhead.
Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
### What changes were proposed in this pull request?
This PR aims to upgrade Scala to 2.12.15 to support Java 17/18 better.
### Why are the changes needed?
Scala 2.12.15 improves compatibility with JDK 17 and 18:
https://github.com/scala/scala/releases/tag/v2.12.15
- Avoids IllegalArgumentException in JDK 17+ for lambda deserialization
- Upgrades to ASM 9.2, for JDK 18 support in optimizer
### Does this PR introduce _any_ user-facing change?
Yes, this is a Scala version change.
### How was this patch tested?
Pass the CIs
Closes#33999 from dongjoon-hyun/SPARK-36759.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request?
Upgrade Apache Parquet to 1.12.1
### Why are the changes needed?
Parquet 1.12.1 contains the following bug fixes:
- PARQUET-2064: Make Range public accessible in RowRanges
- PARQUET-2022: ZstdDecompressorStream should close `zstdInputStream`
- PARQUET-2052: Integer overflow when writing huge binary using dictionary encoding
- PARQUET-1633: Fix integer overflow
- PARQUET-2054: fix TCP leaking when calling ParquetFileWriter.appendFile
- PARQUET-2072: Do Not Determine Both Min/Max for Binary Stats
- PARQUET-2073: Fix estimate remaining row count in ColumnWriteStoreBase
- PARQUET-2078: Failed to read parquet file after writing with the same
In particular PARQUET-2078 is a blocker for the upcoming Apache Spark 3.2.0 release.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests + a new test for the issue in SPARK-36696
Closes#33969 from sunchao/upgrade-parquet-12.1.
Authored-by: Chao Sun <sunchao@apple.com>
Signed-off-by: DB Tsai <d_tsai@apple.com>
### What changes were proposed in this pull request?
Fix Series.update with another in same frame
also add test for update series in diff frame
### Why are the changes needed?
Fix Series.update with another in same frame
Pandas behavior:
``` python
>>> pdf = pd.DataFrame(
... {"a": [None, 2, 3, 4, 5, 6, 7, 8, None], "b": [None, 5, None, 3, 2, 1, None, 0, 0]},
... )
>>> pdf
a b
0 NaN NaN
1 2.0 5.0
2 3.0 NaN
3 4.0 3.0
4 5.0 2.0
5 6.0 1.0
6 7.0 NaN
7 8.0 0.0
8 NaN 0.0
>>> pdf.a.update(pdf.b)
>>> pdf
a b
0 NaN NaN
1 5.0 5.0
2 3.0 NaN
3 3.0 3.0
4 2.0 2.0
5 1.0 1.0
6 7.0 NaN
7 0.0 0.0
8 0.0 0.0
```
### Does this PR introduce _any_ user-facing change?
Before
```python
>>> psdf = ps.DataFrame(
... {"a": [None, 2, 3, 4, 5, 6, 7, 8, None], "b": [None, 5, None, 3, 2, 1, None, 0, 0]},
... )
>>> psdf.a.update(psdf.b)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/dgd/spark/python/pyspark/pandas/series.py", line 4551, in update
combined = combine_frames(self._psdf, other._psdf, how="leftouter")
File "/Users/dgd/spark/python/pyspark/pandas/utils.py", line 141, in combine_frames
assert not same_anchor(
AssertionError: We don't need to combine. `this` and `that` are same.
>>>
```
After
```python
>>> psdf = ps.DataFrame(
... {"a": [None, 2, 3, 4, 5, 6, 7, 8, None], "b": [None, 5, None, 3, 2, 1, None, 0, 0]},
... )
>>> psdf.a.update(psdf.b)
>>> psdf
a b
0 NaN NaN
1 5.0 5.0
2 3.0 NaN
3 3.0 3.0
4 2.0 2.0
5 1.0 1.0
6 7.0 NaN
7 0.0 0.0
8 0.0 0.0
>>>
```
### How was this patch tested?
unit tests
Closes#33968 from dgd-contributor/SPARK-36722_fix_update_same_anchor.
Authored-by: dgd-contributor <dgd_contributor@viettel.com.vn>
Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
### What changes were proposed in this pull request?
For query
```
select arrays_overlap(array(cast('nan' as double), 1d), array(cast('nan' as double)))
```
This returns [false], but it should return [true].
This issue is caused by `scala.mutable.HashSet` can't handle `Double.NaN` and `Float.NaN`.
### Why are the changes needed?
Fix bug
### Does this PR introduce _any_ user-facing change?
arrays_overlap won't handle equal `NaN` value
### How was this patch tested?
Added UT
Closes#34006 from AngersZhuuuu/SPARK-36755.
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
According to https://github.com/apache/spark/pull/33955#discussion_r708570515 use normalized NaN
### Why are the changes needed?
Use normalized NaN for duplicated NaN value
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Exiting UT
Closes#34003 from AngersZhuuuu/SPARK-36702-FOLLOWUP.
Authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
octet_length: caliculate the byte length of strings
bit_length: caliculate the bit length of strings
Those two string related functions are only implemented on SparkSQL, not on Scala, Python and R.
### Why are the changes needed?
Those functions would be useful for multi-bytes character users, who mainly working with Scala, Python or R.
### Does this PR introduce _any_ user-facing change?
Yes. Users can call octet_length/bit_length APIs on Scala(Dataframe), Python, and R.
### How was this patch tested?
unit tests
Closes#33992 from yoda-mon/add-bit-octet-length.
Authored-by: Leona Yoda <yodal@oss.nttdata.com>
Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>
### What changes were proposed in this pull request?
Added cot to pyspark.sql.rst (follow-up)
### Why are the changes needed?
[My previous PR](https://github.com/apache/spark/pull/33906) was missing it.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
manual check
Closes#34002 from yutoacts/SPARK-36660.
Authored-by: Yuto Akutsu <yuto.akutsu@oss.nttdata.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
This PR fixes a perf issue in `SchemaPruning` when a struct has many fields (e.g. >10K fields).
The root cause is `SchemaPruning.sortLeftFieldsByRight` does N * M order searching.
```
val filteredRightFieldNames = rightStruct.fieldNames
.filter(name => leftStruct.fieldNames.exists(resolver(_, name)))
```
To fix this issue, this PR proposes to use `HashMap` to expect a constant order searching.
This PR also adds `case _ if left == right => left` to the method as a short-circuit code.
### Why are the changes needed?
To fix a perf issue.
### Does this PR introduce _any_ user-facing change?
No. The logic should be identical.
### How was this patch tested?
I confirmed that the following micro benchmark finishes within a few seconds.
```
import org.apache.spark.sql.catalyst.expressions.SchemaPruning
import org.apache.spark.sql.types._
var struct1 = new StructType()
(1 to 50000).foreach { i =>
struct1 = struct1.add(new StructField(i + "", IntegerType))
}
var struct2 = new StructType()
(50001 to 100000).foreach { i =>
struct2 = struct2.add(new StructField(i + "", IntegerType))
}
SchemaPruning.sortLeftFieldsByRight(struct1, struct2)
SchemaPruning.sortLeftFieldsByRight(struct2, struct2)
```
The correctness should be checked by existing tests.
Closes#33981 from sarutak/improve-schemapruning-performance.
Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
This PR proposes new syntax to specify the index type and name in pandas API on Spark. This is a base work for SPARK-36707.
More specifically, users now can use the type hints when typing as below:
```
pd.DataFrame[int, [int, int]]
pd.DataFrame[pdf.index.dtype, pdf.dtypes]
pd.DataFrame[("index", int), [("id", int), ("A", int)]]
pd.DataFrame[(pdf.index.name, pdf.index.dtype), zip(pdf.columns, pdf.dtypes)]
```
Note that the types of `[("id", int), ("A", int)]` or `("index", int)` are matched to how you provide a compound NumPy type (see also https://numpy.org/doc/stable/user/basics.rec.html#introduction).
Therefore, the syntax will be:
**Without index:**
```
pd.DataFrame[type, type, ...]
pd.DataFrame[name: type, name: type, ...]
pd.DataFrame[dtypes instance]
pd.DataFrame[zip(names, types)]
```
(New) **With index:**
```
pd.DataFrame[index_type, [type, ...]]
pd.DataFrame[(index_name, index_type), [(name, type), ...]]
pd.DataFrame[dtype instance, dtypes instance]
pd.DataFrame[(index_name, index_type), zip(names, types)]
```
### Why are the changes needed?
Currently, there is no way to specify the type hint for index type - the type hints are converted to return type of pandas UDFs internally. Therefore, we always attach default index which degrade performance:
```python
>>> def transform(pdf) -> pd.DataFrame[int, int]:
... pdf['A'] = pdf.id + 1
... return pdf
...
>>> ks.range(5).koalas.apply_batch(transform)
```
```
c0 c1
0 0 1
1 1 2
2 2 3
3 3 4
4 4 5
```
The [default index](https://koalas.readthedocs.io/en/latest/user_guide/options.html#default-index-type) (for the first column that looks unnamed) is attached when the type hint is specified. For better performance, we should have a way to work around, see also https://github.com/apache/spark/pull/33954#issuecomment-917742920 and [Specify the index column in conversion from Spark DataFrame to Koalas DataFrame](https://koalas.readthedocs.io/en/latest/user_guide/best_practices.html#specify-the-index-column-in-conversion-from-spark-dataframe-to-koalas-dataframe).
Note that this still remains as experimental because Python itself yet doesn't support such kind of typing out of the box. Once pandas completes typing support like NumPy did in `numpy.typing`, we should implement Koalas typing package, and migrate to it with leveraging pandas' typing way.
### Does this PR introduce _any_ user-facing change?
No, this PR does not yet affect any user-facing behavior in theory.
### How was this patch tested?
Unittests were added.
Closes#33954 from HyukjinKwon/SPARK-36709.
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
Upgrade flake8 to 3.8.0 or above in Jenkins
### Why are the changes needed?
In flake8 < 3.8.0, F401 error occurs for imports in if statements when TYPE_CHECKING is True. However, TYPE_CHECKING is always False at runtime, so there is no need to treat it as an error in static analysis.
Since this behavior is fixed In flake8 >= 3.8.0, we should upgrade the flake8 installed in Jenkins to 3.8.0 or above. Otherwise, it occurs F401 error for several lines in pandas-on-PySpark that use TYPE_CHECKING
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass the CI
Closes#32749 from pingsutw/SPARK-34943.
Lead-authored-by: Kevin Su <pingsutw@gmail.com>
Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
This PR aims to fix the regex to avoid breaking `pom.xml`.
### Why are the changes needed?
**BEFORE**
```
$ dev/change-scala-version.sh 2.12
$ git diff | head -n10
diff --git a/core/pom.xml b/core/pom.xml
index dbde22f2bf..6ed368353b 100644
--- a/core/pom.xml
+++ b/core/pom.xml
-35,7 +35,7
</properties>
<dependencies>
- <!--<!--
```
**AFTER**
Since the default Scala version is `2.12`, the following `no-op` is the correct behavior which is consistent with the previous behavior.
```
$ dev/change-scala-version.sh 2.12
$ git diff
```
### Does this PR introduce _any_ user-facing change?
No. This is a dev only change.
### How was this patch tested?
Manually.
Closes#33996 from dongjoon-hyun/SPARK-36712.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request?
SPARK-36456 change to use `JavaUtils. closeQuietly` instead of `IOUtils.closeQuietly`, but there is slightly different from the 2 methods in default behavior: swallowing IOException is same, but the former logs it as ERROR while the latter doesn't log by default.
`Apache commons-io` community decided to retain the `IOUtils.closeQuietly` method in the [new version](75f20dca72/src/main/java/org/apache/commons/io/IOUtils.java (L465-L467)) and removed deprecated annotation, the change has been released in version 2.11.0.
So the change of this pr is to upgrade `Apache commons-io` to 2.11.0 and revert change of SPARK-36456 to maintain original behavior(don't print error log).
### Why are the changes needed?
1. Upgrade `Apache commons-io` to 2.11.0 to use non-deprecated `closeQuietly` API, other changes related to `Apache commons-io are detailed in [commons-io/changes-report](https://commons.apache.org/proper/commons-io/changes-report.html#a2.11.0)
2. Revert change of SPARK-36737 to maintain original `IOUtils.closeQuietly` API behavior(don't print error log).
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass the Jenkins or GitHub Action
Closes#33977 from LuciferYang/upgrade-commons-io.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
### What changes were proposed in this pull request?
For query
```
select array_union(array(cast('nan' as double), cast('nan' as double)), array())
```
This returns [NaN, NaN], but it should return [NaN].
This issue is caused by `OpenHashSet` can't handle `Double.NaN` and `Float.NaN` too.
In this pr we add a wrap for OpenHashSet that can handle `null`, `Double.NaN`, `Float.NaN` together
### Why are the changes needed?
Fix bug
### Does this PR introduce _any_ user-facing change?
ArrayUnion won't show duplicated `NaN` value
### How was this patch tested?
Added UT
Closes#33955 from AngersZhuuuu/SPARK-36702-WrapOpenHashSet.
Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com>
Co-authored-by: AngersZhuuuu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
### What changes were proposed in this pull request?
Only throw logWarning when `PUSH_BASED_SHUFFLE_ENABLED` is set to true and `canDoPushBasedShuffle` is false
### Why are the changes needed?
Currently, this logWarning will still be printed out even when `PUSH_BASED_SHUFFLE_ENABLED` is set to false, which is unnecessary.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Passed existing UT.
Closes#33984 from rmcyang/SPARK-36705-follow-up.
Authored-by: Minchu Yang <minyang@minyang-mn3.linkedin.biz>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
### What changes were proposed in this pull request?
Introduce the 'compute.isin_limit' option, with the default value of 80.
### Why are the changes needed?
`Column.isin(list)` doesn't perform well when the given `list` is large, as https://issues.apache.org/jira/browse/SPARK-33383.
Thus, 'compute.isin_limit' is introduced to constrain the usage of `Column.isin(list)` in the code base.
If the length of the ‘list’ is above the `'compute.isin_limit'`, broadcast join is used instead for better performance.
#### Why is the default value 80?
After reproducing the benchmark mentioned in https://issues.apache.org/jira/browse/SPARK-33383,
| length of filtering list | isin time /ms| broadcast DF time / ms|
| :---: | :-: | :-: |
| 200 | 69411 | 39296 |
| 100 | 43074 | 40087 |
| 80 | 35592 | 40350 |
| 50 | 28134 | 37847 |
We may notice when the length of the filtering list <= 80, the `isin` approach performs better than `broadcast DF`.
### Does this PR introduce _any_ user-facing change?
Users may read/write the value of `'compute.isin_limit'` as follows
```py
>>> ps.get_option('compute.isin_limit')
80
>>> ps.set_option('compute.isin_limit', 10)
>>> ps.get_option('compute.isin_limit')
10
>>> ps.set_option('compute.isin_limit', -1)
...
ValueError: 'compute.isin_limit' should be greater than or equal to 0.
>>> ps.reset_option('compute.isin_limit')
>>> ps.get_option('compute.isin_limit')
80
```
### How was this patch tested?
Manual test.
Closes#33982 from xinrong-databricks/new_option.
Authored-by: Xinrong Meng <xinrong.meng@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
### What changes were proposed in this pull request?
Fix InferFiltersFromGenerate bug, InferFiltersFromGenerate should not infer filter for generate when the children contain an expression which is instance of `org.apache.spark.sql.catalyst.expressions.UserDefinedExpression`.
Before this pr, the following case will throw an exception.
```scala
spark.udf.register("vec", (i: Int) => (0 until i).toArray)
sql("select explode(vec(8)) as c1").show
```
```
Once strategy's idempotence is broken for batch Infer Filters
GlobalLimit 21 GlobalLimit 21
+- LocalLimit 21 +- LocalLimit 21
+- Project [cast(c1#3 as string) AS c1#12] +- Project [cast(c1#3 as string) AS c1#12]
+- Generate explode(vec(8)), false, [c1#3] +- Generate explode(vec(8)), false, [c1#3]
+- Filter ((size(vec(8), true) > 0) AND isnotnull(vec(8))) +- Filter ((size(vec(8), true) > 0) AND isnotnull(vec(8)))
! +- OneRowRelation +- Filter ((size(vec(8), true) > 0) AND isnotnull(vec(8)))
! +- OneRowRelation
java.lang.RuntimeException:
Once strategy's idempotence is broken for batch Infer Filters
GlobalLimit 21 GlobalLimit 21
+- LocalLimit 21 +- LocalLimit 21
+- Project [cast(c1#3 as string) AS c1#12] +- Project [cast(c1#3 as string) AS c1#12]
+- Generate explode(vec(8)), false, [c1#3] +- Generate explode(vec(8)), false, [c1#3]
+- Filter ((size(vec(8), true) > 0) AND isnotnull(vec(8))) +- Filter ((size(vec(8), true) > 0) AND isnotnull(vec(8)))
! +- OneRowRelation +- Filter ((size(vec(8), true) > 0) AND isnotnull(vec(8)))
! +- OneRowRelation
at org.apache.spark.sql.errors.QueryExecutionErrors$.onceStrategyIdempotenceIsBrokenForBatchError(QueryExecutionErrors.scala:1200)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.checkBatchIdempotence(RuleExecutor.scala:168)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:254)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:200)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:200)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:179)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:179)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$optimizedPlan$1(QueryExecution.scala:138)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:196)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:196)
at org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:134)
at org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:130)
at org.apache.spark.sql.execution.QueryExecution.assertOptimized(QueryExecution.scala:148)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executedPlan$1(QueryExecution.scala:166)
at org.apache.spark.sql.execution.QueryExecution.withCteMap(QueryExecution.scala:73)
at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:163)
at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:163)
at org.apache.spark.sql.execution.QueryExecution.simpleString(QueryExecution.scala:214)
at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$explainString(QueryExecution.scala:259)
at org.apache.spark.sql.execution.QueryExecution.explainString(QueryExecution.scala:228)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3731)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2755)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2962)
at org.apache.spark.sql.Dataset.getRows(Dataset.scala:288)
at org.apache.spark.sql.Dataset.showString(Dataset.scala:327)
at org.apache.spark.sql.Dataset.show(Dataset.scala:807)
```
### Does this PR introduce _any_ user-facing change?
No, only bug fix.
### How was this patch tested?
Unit test.
Closes#33956 from cfmcgrady/SPARK-36715.
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
