[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
HikariCP-java7-2.4.12.jar
|
|
|
|
|
JavaEWAH-0.3.2.jar
|
|
|
|
|
RoaringBitmap-0.5.11.jar
|
|
|
|
|
ST4-4.0.4.jar
|
|
|
|
|
accessors-smart-1.2.jar
|
|
|
|
|
activation-1.1.1.jar
|
2018-07-18 02:52:17 -04:00
|
|
|
aircompressor-0.10.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
antlr-2.7.7.jar
|
|
|
|
|
antlr-runtime-3.4.jar
|
|
|
|
|
antlr4-runtime-4.7.jar
|
|
|
|
|
aopalliance-1.0.jar
|
|
|
|
|
aopalliance-repackaged-2.4.0-b34.jar
|
|
|
|
|
apache-log4j-extras-1.2.17.jar
|
|
|
|
|
arpack_combined_all-0.1.jar
|
|
|
|
|
arrow-format-0.8.0.jar
|
|
|
|
|
arrow-memory-0.8.0.jar
|
|
|
|
|
arrow-vector-0.8.0.jar
|
|
|
|
|
automaton-1.11-8.jar
|
|
|
|
|
avro-1.7.7.jar
|
|
|
|
|
avro-ipc-1.7.7.jar
|
|
|
|
|
avro-mapred-1.7.7-hadoop2.jar
|
|
|
|
|
base64-2.3.8.jar
|
|
|
|
|
bcprov-jdk15on-1.58.jar
|
|
|
|
|
bonecp-0.8.0.RELEASE.jar
|
|
|
|
|
breeze-macros_2.11-0.13.2.jar
|
|
|
|
|
breeze_2.11-0.13.2.jar
|
|
|
|
|
calcite-avatica-1.2.0-incubating.jar
|
|
|
|
|
calcite-core-1.2.0-incubating.jar
|
|
|
|
|
calcite-linq4j-1.2.0-incubating.jar
|
|
|
|
|
chill-java-0.8.4.jar
|
|
|
|
|
chill_2.11-0.8.4.jar
|
|
|
|
|
commons-beanutils-1.9.3.jar
|
|
|
|
|
commons-cli-1.2.jar
|
|
|
|
|
commons-codec-1.10.jar
|
|
|
|
|
commons-collections-3.2.2.jar
|
|
|
|
|
commons-compiler-3.0.8.jar
|
|
|
|
|
commons-compress-1.4.1.jar
|
|
|
|
|
commons-configuration2-2.1.1.jar
|
|
|
|
|
commons-crypto-1.0.0.jar
|
|
|
|
|
commons-daemon-1.0.13.jar
|
|
|
|
|
commons-dbcp-1.4.jar
|
|
|
|
|
commons-httpclient-3.1.jar
|
|
|
|
|
commons-io-2.4.jar
|
|
|
|
|
commons-lang-2.6.jar
|
|
|
|
|
commons-lang3-3.5.jar
|
|
|
|
|
commons-logging-1.1.3.jar
|
|
|
|
|
commons-math3-3.4.1.jar
|
|
|
|
|
commons-net-3.1.jar
|
|
|
|
|
commons-pool-1.5.4.jar
|
|
|
|
|
compress-lzf-1.0.3.jar
|
|
|
|
|
core-1.1.2.jar
|
|
|
|
|
curator-client-2.12.0.jar
|
|
|
|
|
curator-framework-2.12.0.jar
|
|
|
|
|
curator-recipes-2.12.0.jar
|
|
|
|
|
datanucleus-api-jdo-3.2.6.jar
|
|
|
|
|
datanucleus-core-3.2.10.jar
|
|
|
|
|
datanucleus-rdbms-3.2.9.jar
|
|
|
|
|
derby-10.12.1.1.jar
|
|
|
|
|
dnsjava-2.1.7.jar
|
|
|
|
|
ehcache-3.3.1.jar
|
|
|
|
|
eigenbase-properties-1.1.5.jar
|
|
|
|
|
flatbuffers-1.2.0-3f79e055.jar
|
|
|
|
|
generex-1.0.1.jar
|
|
|
|
|
geronimo-jcache_1.0_spec-1.0-alpha-1.jar
|
|
|
|
|
gson-2.2.4.jar
|
|
|
|
|
guava-14.0.1.jar
|
|
|
|
|
guice-4.0.jar
|
|
|
|
|
guice-servlet-4.0.jar
|
|
|
|
|
hadoop-annotations-3.1.0.jar
|
|
|
|
|
hadoop-auth-3.1.0.jar
|
|
|
|
|
hadoop-client-3.1.0.jar
|
|
|
|
|
hadoop-common-3.1.0.jar
|
|
|
|
|
hadoop-hdfs-client-3.1.0.jar
|
|
|
|
|
hadoop-mapreduce-client-common-3.1.0.jar
|
|
|
|
|
hadoop-mapreduce-client-core-3.1.0.jar
|
|
|
|
|
hadoop-mapreduce-client-jobclient-3.1.0.jar
|
|
|
|
|
hadoop-yarn-api-3.1.0.jar
|
|
|
|
|
hadoop-yarn-client-3.1.0.jar
|
|
|
|
|
hadoop-yarn-common-3.1.0.jar
|
|
|
|
|
hadoop-yarn-registry-3.1.0.jar
|
|
|
|
|
hadoop-yarn-server-common-3.1.0.jar
|
|
|
|
|
hadoop-yarn-server-web-proxy-3.1.0.jar
|
|
|
|
|
hk2-api-2.4.0-b34.jar
|
|
|
|
|
hk2-locator-2.4.0-b34.jar
|
|
|
|
|
hk2-utils-2.4.0-b34.jar
|
|
|
|
|
hppc-0.7.2.jar
|
|
|
|
|
htrace-core4-4.1.0-incubating.jar
|
|
|
|
|
httpclient-4.5.4.jar
|
|
|
|
|
httpcore-4.4.8.jar
|
|
|
|
|
ivy-2.4.0.jar
|
|
|
|
|
jackson-annotations-2.6.7.jar
|
|
|
|
|
jackson-core-2.6.7.jar
|
|
|
|
|
jackson-core-asl-1.9.13.jar
|
|
|
|
|
jackson-databind-2.6.7.1.jar
|
|
|
|
|
jackson-dataformat-yaml-2.6.7.jar
|
|
|
|
|
jackson-jaxrs-base-2.7.8.jar
|
|
|
|
|
jackson-jaxrs-json-provider-2.7.8.jar
|
|
|
|
|
jackson-mapper-asl-1.9.13.jar
|
|
|
|
|
jackson-module-jaxb-annotations-2.6.7.jar
|
|
|
|
|
jackson-module-paranamer-2.7.9.jar
|
|
|
|
|
jackson-module-scala_2.11-2.6.7.1.jar
|
|
|
|
|
janino-3.0.8.jar
|
|
|
|
|
java-xmlbuilder-1.1.jar
|
|
|
|
|
javassist-3.18.1-GA.jar
|
|
|
|
|
javax.annotation-api-1.2.jar
|
|
|
|
|
javax.inject-1.jar
|
|
|
|
|
javax.inject-2.4.0-b34.jar
|
|
|
|
|
javax.servlet-api-3.1.0.jar
|
|
|
|
|
javax.ws.rs-api-2.0.1.jar
|
|
|
|
|
javolution-5.5.1.jar
|
|
|
|
|
jaxb-api-2.2.11.jar
|
|
|
|
|
jcip-annotations-1.0-1.jar
|
|
|
|
|
jcl-over-slf4j-1.7.16.jar
|
|
|
|
|
jdo-api-3.0.1.jar
|
|
|
|
|
jersey-client-2.22.2.jar
|
|
|
|
|
jersey-common-2.22.2.jar
|
|
|
|
|
jersey-container-servlet-2.22.2.jar
|
|
|
|
|
jersey-container-servlet-core-2.22.2.jar
|
|
|
|
|
jersey-guava-2.22.2.jar
|
|
|
|
|
jersey-media-jaxb-2.22.2.jar
|
|
|
|
|
jersey-server-2.22.2.jar
|
|
|
|
|
jets3t-0.9.4.jar
|
|
|
|
|
jetty-webapp-9.3.20.v20170531.jar
|
|
|
|
|
jetty-xml-9.3.20.v20170531.jar
|
[SPARK-24418][BUILD] Upgrade Scala to 2.11.12 and 2.12.6
## What changes were proposed in this pull request?
Scala is upgraded to `2.11.12` and `2.12.6`.
We used `loadFIles()` in `ILoop` as a hook to initialize the Spark before REPL sees any files in Scala `2.11.8`. However, it was a hack, and it was not intended to be a public API, so it was removed in Scala `2.11.12`.
From the discussion in Scala community, https://github.com/scala/bug/issues/10913 , we can use `initializeSynchronous` to initialize Spark instead. This PR implements the Spark initialization there.
However, in Scala `2.11.12`'s `ILoop.scala`, in function `def startup()`, the first thing it calls is `printWelcome()`. As a result, Scala will call `printWelcome()` and `splash` before calling `initializeSynchronous`.
Thus, the Spark shell will allow users to type commends first, and then show the Spark UI URL. It's working, but it will change the Spark Shell interface as the following.
```scala
➜ apache-spark git:(scala-2.11.12) ✗ ./bin/spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0-SNAPSHOT
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)
Type in expressions to have them evaluated.
Type :help for more information.
scala> Spark context Web UI available at http://192.168.1.169:4040
Spark context available as 'sc' (master = local[*], app id = local-1528180279528).
Spark session available as 'spark'.
scala>
```
It seems there is no easy way to inject the Spark initialization code in the proper place as Scala doesn't provide a hook. Maybe som-snytt can comment on this.
The following command is used to update the dep files.
```scala
./dev/test-dependencies.sh --replace-manifest
```
## How was this patch tested?
Existing tests
Author: DB Tsai <d_tsai@apple.com>
Closes #21495 from dbtsai/scala-2.11.12.
2018-06-25 21:48:52 -04:00
|
|
|
jline-2.14.3.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
joda-time-2.9.3.jar
|
|
|
|
|
jodd-core-3.5.2.jar
|
|
|
|
|
jpam-1.1.jar
|
|
|
|
|
json-smart-2.3.jar
|
|
|
|
|
json4s-ast_2.11-3.5.3.jar
|
|
|
|
|
json4s-core_2.11-3.5.3.jar
|
|
|
|
|
json4s-jackson_2.11-3.5.3.jar
|
|
|
|
|
json4s-scalap_2.11-3.5.3.jar
|
|
|
|
|
jsp-api-2.1.jar
|
|
|
|
|
jsr305-1.3.9.jar
|
|
|
|
|
jta-1.1.jar
|
|
|
|
|
jtransforms-2.4.0.jar
|
|
|
|
|
jul-to-slf4j-1.7.16.jar
|
|
|
|
|
kerb-admin-1.0.1.jar
|
|
|
|
|
kerb-client-1.0.1.jar
|
|
|
|
|
kerb-common-1.0.1.jar
|
|
|
|
|
kerb-core-1.0.1.jar
|
|
|
|
|
kerb-crypto-1.0.1.jar
|
|
|
|
|
kerb-identity-1.0.1.jar
|
|
|
|
|
kerb-server-1.0.1.jar
|
|
|
|
|
kerb-simplekdc-1.0.1.jar
|
|
|
|
|
kerb-util-1.0.1.jar
|
|
|
|
|
kerby-asn1-1.0.1.jar
|
|
|
|
|
kerby-config-1.0.1.jar
|
|
|
|
|
kerby-pkix-1.0.1.jar
|
|
|
|
|
kerby-util-1.0.1.jar
|
|
|
|
|
kerby-xdr-1.0.1.jar
|
|
|
|
|
kryo-shaded-3.0.3.jar
|
|
|
|
|
kubernetes-client-3.0.0.jar
|
|
|
|
|
kubernetes-model-2.0.0.jar
|
|
|
|
|
leveldbjni-all-1.8.jar
|
|
|
|
|
libfb303-0.9.3.jar
|
|
|
|
|
libthrift-0.9.3.jar
|
|
|
|
|
log4j-1.2.17.jar
|
|
|
|
|
logging-interceptor-3.8.1.jar
|
|
|
|
|
lz4-java-1.4.0.jar
|
|
|
|
|
machinist_2.11-0.6.1.jar
|
|
|
|
|
macro-compat_2.11-1.1.1.jar
|
|
|
|
|
mesos-1.4.0-shaded-protobuf.jar
|
|
|
|
|
metrics-core-3.1.5.jar
|
|
|
|
|
metrics-graphite-3.1.5.jar
|
|
|
|
|
metrics-json-3.1.5.jar
|
|
|
|
|
metrics-jvm-3.1.5.jar
|
|
|
|
|
minlog-1.3.0.jar
|
|
|
|
|
mssql-jdbc-6.2.1.jre7.jar
|
|
|
|
|
netty-3.9.9.Final.jar
|
|
|
|
|
netty-all-4.1.17.Final.jar
|
|
|
|
|
nimbus-jose-jwt-4.41.1.jar
|
|
|
|
|
objenesis-2.1.jar
|
|
|
|
|
okhttp-2.7.5.jar
|
|
|
|
|
okhttp-3.8.1.jar
|
|
|
|
|
okio-1.13.0.jar
|
|
|
|
|
opencsv-2.3.jar
|
2018-07-18 02:52:17 -04:00
|
|
|
orc-core-1.5.2-nohive.jar
|
|
|
|
|
orc-mapreduce-1.5.2-nohive.jar
|
|
|
|
|
orc-shims-1.5.2.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
oro-2.0.8.jar
|
|
|
|
|
osgi-resource-locator-1.0.1.jar
|
|
|
|
|
paranamer-2.8.jar
|
2018-05-09 00:27:32 -04:00
|
|
|
parquet-column-1.10.0.jar
|
|
|
|
|
parquet-common-1.10.0.jar
|
|
|
|
|
parquet-encoding-1.10.0.jar
|
|
|
|
|
parquet-format-2.4.0.jar
|
|
|
|
|
parquet-hadoop-1.10.0.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
parquet-hadoop-bundle-1.6.0.jar
|
2018-05-09 00:27:32 -04:00
|
|
|
parquet-jackson-1.10.0.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
protobuf-java-2.5.0.jar
|
2018-04-13 17:28:24 -04:00
|
|
|
py4j-0.10.7.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
pyrolite-4.13.jar
|
|
|
|
|
re2j-1.1.jar
|
[SPARK-24418][BUILD] Upgrade Scala to 2.11.12 and 2.12.6
## What changes were proposed in this pull request?
Scala is upgraded to `2.11.12` and `2.12.6`.
We used `loadFIles()` in `ILoop` as a hook to initialize the Spark before REPL sees any files in Scala `2.11.8`. However, it was a hack, and it was not intended to be a public API, so it was removed in Scala `2.11.12`.
From the discussion in Scala community, https://github.com/scala/bug/issues/10913 , we can use `initializeSynchronous` to initialize Spark instead. This PR implements the Spark initialization there.
However, in Scala `2.11.12`'s `ILoop.scala`, in function `def startup()`, the first thing it calls is `printWelcome()`. As a result, Scala will call `printWelcome()` and `splash` before calling `initializeSynchronous`.
Thus, the Spark shell will allow users to type commends first, and then show the Spark UI URL. It's working, but it will change the Spark Shell interface as the following.
```scala
➜ apache-spark git:(scala-2.11.12) ✗ ./bin/spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0-SNAPSHOT
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)
Type in expressions to have them evaluated.
Type :help for more information.
scala> Spark context Web UI available at http://192.168.1.169:4040
Spark context available as 'sc' (master = local[*], app id = local-1528180279528).
Spark session available as 'spark'.
scala>
```
It seems there is no easy way to inject the Spark initialization code in the proper place as Scala doesn't provide a hook. Maybe som-snytt can comment on this.
The following command is used to update the dep files.
```scala
./dev/test-dependencies.sh --replace-manifest
```
## How was this patch tested?
Existing tests
Author: DB Tsai <d_tsai@apple.com>
Closes #21495 from dbtsai/scala-2.11.12.
2018-06-25 21:48:52 -04:00
|
|
|
scala-compiler-2.11.12.jar
|
|
|
|
|
scala-library-2.11.12.jar
|
|
|
|
|
scala-parser-combinators_2.11-1.1.0.jar
|
|
|
|
|
scala-reflect-2.11.12.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
scala-xml_2.11-1.0.5.jar
|
|
|
|
|
shapeless_2.11-2.3.2.jar
|
|
|
|
|
slf4j-api-1.7.16.jar
|
|
|
|
|
slf4j-log4j12-1.7.16.jar
|
|
|
|
|
snakeyaml-1.15.jar
|
|
|
|
|
snappy-0.2.jar
|
|
|
|
|
snappy-java-1.1.7.1.jar
|
|
|
|
|
spire-macros_2.11-0.13.0.jar
|
|
|
|
|
spire_2.11-0.13.0.jar
|
|
|
|
|
stax-api-1.0.1.jar
|
|
|
|
|
stax2-api-3.1.4.jar
|
|
|
|
|
stream-2.7.0.jar
|
|
|
|
|
stringtemplate-3.2.1.jar
|
|
|
|
|
super-csv-2.2.0.jar
|
|
|
|
|
token-provider-1.0.1.jar
|
2018-05-13 22:01:06 -04:00
|
|
|
univocity-parsers-2.6.3.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
validation-api-1.1.0.Final.jar
|
|
|
|
|
woodstox-core-5.0.3.jar
|
2018-07-03 13:13:48 -04:00
|
|
|
xbean-asm6-shaded-4.8.jar
|
[SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevant POM fix ups
## What changes were proposed in this pull request?
1. Adds a `hadoop-3.1` profile build depending on the hadoop-3.1 artifacts.
1. In the hadoop-cloud module, adds an explicit hadoop-3.1 profile which switches from explicitly pulling in cloud connectors (hadoop-openstack, hadoop-aws, hadoop-azure) to depending on the hadoop-cloudstorage POM artifact, which pulls these in, has pre-excluded things like hadoop-common, and stays up to date with new connectors (hadoop-azuredatalake, hadoop-allyun). Goal: it becomes the Hadoop projects homework of keeping this clean, and the spark project doesn't need to handle new hadoop releases adding more dependencies.
1. the hadoop-cloud/hadoop-3.1 profile also declares support for jetty-ajax and jetty-util to ensure that these jars get into the distribution jar directory when needed by unshaded libraries.
1. Increases the curator and zookeeper versions to match those in hadoop-3, fixing spark core to build in sbt with the hadoop-3 dependencies.
## How was this patch tested?
* Everything this has been built and tested against both ASF Hadoop branch-3.1 and hadoop trunk.
* spark-shell was used to create connectors to all the stores and verify that file IO could take place.
The spark hive-1.2.1 JAR has problems here, as it's version check logic fails for Hadoop versions > 2.
This can be avoided with either of
* The hadoop JARs built to declare their version as Hadoop 2.11 `mvn install -DskipTests -DskipShade -Ddeclared.hadoop.version=2.11` . This is safe for local test runs, not for deployment (HDFS is very strict about cross-version deployment).
* A modified version of spark hive whose version check switch statement is happy with hadoop 3.
I've done both, with maven and SBT.
Three issues surfaced
1. A spark-core test failure —fixed in SPARK-23787.
1. SBT only: Zookeeper not being found in spark-core. Somehow curator 2.12.0 triggers some slightly different dependency resolution logic from previous versions, and Ivy was missing zookeeper.jar entirely. This patch adds the explicit declaration for all spark profiles, setting the ZK version = 3.4.9 for hadoop-3.1
1. Marking jetty-utils as provided in spark was stopping hadoop-azure from being able to instantiate the azure wasb:// client; it was using jetty-util-ajax, which could then not find a class in jetty-util.
Author: Steve Loughran <stevel@hortonworks.com>
Closes #20923 from steveloughran/cloud/SPARK-23807-hadoop-31.
2018-04-24 12:57:09 -04:00
|
|
|
xz-1.0.jar
|
|
|
|
|
zjsonpatch-0.3.0.jar
|
|
|
|
|
zookeeper-3.4.9.jar
|
|
|
|
|
zstd-jni-1.3.2-2.jar
|
