spark-instrumented-optimizer/streaming
yangjie01 dd80845735 [SPARK-32964][DSTREAMS] Pass all streaming module UTs in Scala 2.13
### What changes were proposed in this pull request?

There is only one failed case of `streaming` module in Scala 2.13: `start with non-serializable DStream checkpoint ` in `StreamingContextSuite`.

`StackOverflowError` is thrown here when `SerializationDebugger#visit` method is called.

I found that `inputStreams` and `outputStreams` in `DStreamGraph` can not be matched in `SerializationDebugger#visit` method because `ArrayBuffer` in not `Array` in Scala 2.13.

The main change of this pr is use `mutable.ArraySeq` instead of `ArrayBuffer` to store `inputStreams` and `outputStreams` in `DStreamGraph`, then it can be matched in `SerializationDebugger#visit` method.

### Why are the changes needed?
We need to support a Scala 2.13 build.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

- Scala 2.12: Pass the Jenkins or GitHub Action

- Scala 2.13: Pass GitHub 2.13 Build Action

Do the following:

```
dev/change-scala-version.sh 2.13
mvn clean install -DskipTests  -pl streaming -Pscala-2.13 -am
mvn test -pl streaming -Pscala-2.13
mvn test -pl core -Pscala-2.13
```

streaming module:

```
Tests: succeeded 339, failed 0, canceled 0, ignored 2, pending 0
All tests passed.
```

core module:

```
Tests: succeeded 2648, failed 0, canceled 4, ignored 7, pending 0
All tests passed.
```

Closes #29836 from LuciferYang/fix-streaming-213.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
2020-09-22 11:01:44 -07:00
..
src [SPARK-32964][DSTREAMS] Pass all streaming module UTs in Scala 2.13 2020-09-22 11:01:44 -07:00
pom.xml [SPARK-31765][WEBUI][TEST-MAVEN] Upgrade HtmlUnit >= 2.37.0 2020-06-11 18:27:53 -05:00