689386b1c6
This patch modifies Spark's SBT build so that it no longer uses `retrieveManaged` / `lib_managed` to store its dependencies. The motivations for this change are nicely described on the JIRA ticket ([SPARK-7841](https://issues.apache.org/jira/browse/SPARK-7841)); my personal interest in doing this stems from the fact that `lib_managed` has caused me some pain while debugging dependency issues in another PR of mine. Removing our use of `lib_managed` would be trivial except for one snag: the Datanucleus JARs, required by Spark SQL's Hive integration, cannot be included in assembly JARs due to problems with merging OSGI `plugin.xml` files. As a result, several places in the packaging and deployment pipeline assume that these Datanucleus JARs are copied to `lib_managed/jars`. In the interest of maintaining compatibility, I have chosen to retain the `lib_managed/jars` directory _only_ for these Datanucleus JARs and have added custom code to `SparkBuild.scala` to automatically copy those JARs to that folder as part of the `assembly` task. `dev/mima` also depended on `lib_managed` in a hacky way in order to set classpaths when generating MiMa excludes; I've updated this to obtain the classpaths directly from SBT instead. /cc dragos marmbrus pwendell srowen Author: Josh Rosen <joshrosen@databricks.com> Closes #9575 from JoshRosen/SPARK-7841.
55 lines
1.8 KiB
Bash
Executable file
55 lines
1.8 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
set -o pipefail
|
|
set -e
|
|
|
|
# Go to the Spark project root directory
|
|
FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
|
|
cd "$FWDIR"
|
|
|
|
echo -e "q\n" | build/sbt oldDeps/update
|
|
rm -f .generated-mima*
|
|
|
|
generate_mima_ignore() {
|
|
SPARK_JAVA_OPTS="-XX:MaxPermSize=1g -Xmx2g" \
|
|
./bin/spark-class org.apache.spark.tools.GenerateMIMAIgnore
|
|
}
|
|
|
|
# Generate Mima Ignore is called twice, first with latest built jars
|
|
# on the classpath and then again with previous version jars on the classpath.
|
|
# Because of a bug in GenerateMIMAIgnore that when old jars are ahead on classpath
|
|
# it did not process the new classes (which are in assembly jar).
|
|
generate_mima_ignore
|
|
|
|
export SPARK_CLASSPATH="$(build/sbt "export oldDeps/fullClasspath" | tail -n1)"
|
|
echo "SPARK_CLASSPATH=$SPARK_CLASSPATH"
|
|
|
|
generate_mima_ignore
|
|
|
|
echo -e "q\n" | build/sbt mima-report-binary-issues | grep -v -e "info.*Resolving"
|
|
ret_val=$?
|
|
|
|
if [ $ret_val != 0 ]; then
|
|
echo "NOTE: Exceptions to binary compatibility can be added in project/MimaExcludes.scala"
|
|
fi
|
|
|
|
rm -f .generated-mima*
|
|
exit $ret_val
|