12ab7f7e89
…build; fix some things that will be warnings or errors in 2.12; restore Scala 2.12 profile infrastructure ## What changes were proposed in this pull request? This change adds back the infrastructure for a Scala 2.12 build, but does not enable it in the release or Python test scripts. In order to make that meaningful, it also resolves compile errors that the code hits in 2.12 only, in a way that still works with 2.11. It also updates dependencies to the earliest minor release of dependencies whose current version does not yet support Scala 2.12. This is in a sense covered by other JIRAs under the main umbrella, but implemented here. The versions below still work with 2.11, and are the _latest_ maintenance release in the _earliest_ viable minor release. - Scalatest 2.x -> 3.0.3 - Chill 0.8.0 -> 0.8.4 - Clapper 1.0.x -> 1.1.2 - json4s 3.2.x -> 3.4.2 - Jackson 2.6.x -> 2.7.9 (required by json4s) This change does _not_ fully enable a Scala 2.12 build: - It will also require dropping support for Kafka before 0.10. Easy enough, just didn't do it yet here - It will require recreating `SparkILoop` and `Main` for REPL 2.12, which is SPARK-14650. Possible to do here too. What it does do is make changes that resolve much of the remaining gap without affecting the current 2.11 build. ## How was this patch tested? Existing tests and build. Manually tested with `./dev/change-scala-version.sh 2.12` to verify it compiles, modulo the exceptions above. Author: Sean Owen <sowen@cloudera.com> Closes #18645 from srowen/SPARK-14280.
60 lines
1.8 KiB
Batchfile
60 lines
1.8 KiB
Batchfile
@echo off
|
|
|
|
rem
|
|
rem Licensed to the Apache Software Foundation (ASF) under one or more
|
|
rem contributor license agreements. See the NOTICE file distributed with
|
|
rem this work for additional information regarding copyright ownership.
|
|
rem The ASF licenses this file to You under the Apache License, Version 2.0
|
|
rem (the "License"); you may not use this file except in compliance with
|
|
rem the License. You may obtain a copy of the License at
|
|
rem
|
|
rem http://www.apache.org/licenses/LICENSE-2.0
|
|
rem
|
|
rem Unless required by applicable law or agreed to in writing, software
|
|
rem distributed under the License is distributed on an "AS IS" BASIS,
|
|
rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
rem See the License for the specific language governing permissions and
|
|
rem limitations under the License.
|
|
rem
|
|
|
|
rem This script loads spark-env.cmd if it exists, and ensures it is only loaded once.
|
|
rem spark-env.cmd is loaded from SPARK_CONF_DIR if set, or within the current directory's
|
|
rem conf/ subdirectory.
|
|
|
|
if [%SPARK_ENV_LOADED%] == [] (
|
|
set SPARK_ENV_LOADED=1
|
|
|
|
if not [%SPARK_CONF_DIR%] == [] (
|
|
set user_conf_dir=%SPARK_CONF_DIR%
|
|
) else (
|
|
set user_conf_dir=..\conf
|
|
)
|
|
|
|
call :LoadSparkEnv
|
|
)
|
|
|
|
rem Setting SPARK_SCALA_VERSION if not already set.
|
|
|
|
set ASSEMBLY_DIR2="%SPARK_HOME%\assembly\target\scala-2.11"
|
|
set ASSEMBLY_DIR1="%SPARK_HOME%\assembly\target\scala-2.12"
|
|
|
|
if [%SPARK_SCALA_VERSION%] == [] (
|
|
|
|
if exist %ASSEMBLY_DIR2% if exist %ASSEMBLY_DIR1% (
|
|
echo "Presence of build for multiple Scala versions detected."
|
|
echo "Either clean one of them or, set SPARK_SCALA_VERSION in spark-env.cmd."
|
|
exit 1
|
|
)
|
|
if exist %ASSEMBLY_DIR2% (
|
|
set SPARK_SCALA_VERSION=2.11
|
|
) else (
|
|
set SPARK_SCALA_VERSION=2.12
|
|
)
|
|
)
|
|
exit /b 0
|
|
|
|
:LoadSparkEnv
|
|
if exist "%user_conf_dir%\spark-env.cmd" (
|
|
call "%user_conf_dir%\spark-env.cmd"
|
|
)
|