ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-17058][BUILD] Add maven snapshots-and-staging profile to build/test against staging artifacts

Browse source 
## What changes were proposed in this pull request?

Adds a `snapshots-and-staging profile` so that  RCs of projects like Hadoop and HBase can be used in developer-only build and test runs. There's a comment above the profile telling people not to use this in production.

There's no attempt to do the same for SBT, as Ivy is different.
## How was this patch tested?

Tested by building against the Hadoop 2.7.3 RC 1 JARs

without the profile (and without any local copy of the 2.7.3 artifacts), the build failed

```
mvn install -DskipTests -Pyarn,hadoop-2.7,hive -Dhadoop.version=2.7.3

...

[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Launcher 2.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.7.3/hadoop-client-2.7.3.pom
[WARNING] The POM for org.apache.hadoop:hadoop-client:jar:2.7.3 is missing, no dependency information available
Downloading: https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.7.3/hadoop-client-2.7.3.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  4.482 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 17.402 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 11.252 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 13.458 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  9.043 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 16.027 s]
[INFO] Spark Project Launcher ............................. FAILURE [  1.653 s]
[INFO] Spark Project Core ................................. SKIPPED
...
```

With the profile, the build completed

```
mvn install -DskipTests -Pyarn,hadoop-2.7,hive,snapshots-and-staging -Dhadoop.version=2.7.3
```

Author: Steve Loughran <stevel@apache.org>

Closes #14646 from steveloughran/stevel/SPARK-17058-support-asf-snapshots.
This commit is contained in:
Steve Loughran 2016-11-02 11:52:29 -07:00 committed by Reynold Xin
parent 3c24299b71
commit 37d95227a2
1 changed files with 48 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

48
pom.xml
View file

@ -2693,6 +2693,54 @@
</build>
</profile>
<!--
This is a profile to enable the use of the ASF snapshot and staging repositories
during a build. It is useful when testing againt nightly or RC releases of dependencies.
It MUST NOT be used when building copies of Spark to use in production of for distribution,
-->
<profile>
<id>snapshots-and-staging</id>
<properties>
<!-- override point for ASF staging/snapshot repos -->
<asf.staging>https://repository.apache.org/content/groups/staging/</asf.staging>
<asf.snapshots>https://repository.apache.org/content/repositories/snapshots/</asf.snapshots>
</properties>
<pluginRepositories>
<pluginRepository>
<id>ASF Staging</id>
<url>${asf.staging}</url>
</pluginRepository>
<pluginRepository>
<id>ASF Snapshots</id>
<url>${asf.snapshots}</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>false</enabled>
</releases>
</pluginRepository>
</pluginRepositories>
<repositories>
<repository>
<id>ASF Staging</id>
<url>${asf.staging}</url>
</repository>
<repository>
<id>ASF Snapshots</id>
<url>${asf.snapshots}</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>false</enabled>
</releases>
</repository>
</repositories>
</profile>
<!--
These empty profiles are available in some sub-modules. Declare them here so that
maven does not complain when they're provided on the command line for a sub-module