[SPARK-20343][BUILD] Force Avro 1.7.7 in sbt build to resolve build failure in SBT Hadoop 2.6 master on Jenkins
## What changes were proposed in this pull request? This PR proposes to force Avro's version to 1.7.7 in core to resolve the build failure as below: ``` [error] /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData [error] writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema)) [error] ``` https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/2770/consoleFull Note that this is a hack and should be removed in the future. ## How was this patch tested? I only tested this actually overrides the dependency. I tried many ways but I was unable to reproduce this in my local. Sean also tried the way I did but he was also unable to reproduce this. Please refer the comments in https://github.com/apache/spark/pull/17477#issuecomment-294094092 Author: hyukjinkwon <gurwls223@gmail.com> Closes #17651 from HyukjinKwon/SPARK-20343-sbt.
This commit is contained in:
parent
07fd94e0d0
commit
d4f10cbbe1
1
pom.xml
1
pom.xml
|
@ -142,6 +142,7 @@
|
|||
<ivy.version>2.4.0</ivy.version>
|
||||
<oro.version>2.0.8</oro.version>
|
||||
<codahale.metrics.version>3.1.2</codahale.metrics.version>
|
||||
<!-- Keep consistent with Avro vesion in SBT build for SPARK-20343 -->
|
||||
<avro.version>1.7.7</avro.version>
|
||||
<avro.mapred.classifier>hadoop2</avro.mapred.classifier>
|
||||
<jets3t.version>0.9.3</jets3t.version>
|
||||
|
|
|
@ -318,8 +318,8 @@ object SparkBuild extends PomBuild {
|
|||
enable(MimaBuild.mimaSettings(sparkHome, x))(x)
|
||||
}
|
||||
|
||||
/* Generate and pick the spark build info from extra-resources */
|
||||
enable(Core.settings)(core)
|
||||
/* Generate and pick the spark build info from extra-resources and override a dependency */
|
||||
enable(Core.settings ++ CoreDependencyOverrides.settings)(core)
|
||||
|
||||
/* Unsafe settings */
|
||||
enable(Unsafe.settings)(unsafe)
|
||||
|
@ -443,6 +443,16 @@ object DockerIntegrationTests {
|
|||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Overrides to work around sbt's dependency resolution being different from Maven's in Unidoc.
|
||||
*
|
||||
* Note that, this is a hack that should be removed in the future. See SPARK-20343
|
||||
*/
|
||||
object CoreDependencyOverrides {
|
||||
lazy val settings = Seq(
|
||||
dependencyOverrides += "org.apache.avro" % "avro" % "1.7.7")
|
||||
}
|
||||
|
||||
/**
|
||||
* Overrides to work around sbt's dependency resolution being different from Maven's.
|
||||
*/
|
||||
|
|
Loading…
Reference in a new issue