9b792518b2
### What changes were proposed in this pull request? If a Spark distribution has built-in hadoop runtime, Spark will not populate the hadoop classpath from `yarn.application.classpath` and `mapreduce.application.classpath` when a job is submitted to Yarn. Users can override this behavior by setting `spark.yarn.populateHadoopClasspath` to `true`. ### Why are the changes needed? Without this, Spark will populate the hadoop classpath from `yarn.application.classpath` and `mapreduce.application.classpath` even Spark distribution has built-in hadoop. This results jar conflict and many unexpected behaviors in runtime. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually test with two builds, with-hadoop and no-hadoop builds. Closes #28788 from dbtsai/yarn-classpath. Authored-by: DB Tsai <d_tsai@apple.com> Signed-off-by: DB Tsai <d_tsai@apple.com>
127 lines
2 KiB
Plaintext
127 lines
2 KiB
Plaintext
target
|
|
cache
|
|
.gitignore
|
|
.gitattributes
|
|
.project
|
|
.classpath
|
|
.mima-excludes
|
|
.generated-mima-excludes
|
|
.generated-mima-class-excludes
|
|
.generated-mima-member-excludes
|
|
.rat-excludes
|
|
.*md
|
|
derby.log
|
|
licenses/*
|
|
licenses-binary/*
|
|
LICENSE
|
|
NOTICE
|
|
TAGS
|
|
RELEASE
|
|
control
|
|
docs
|
|
slaves
|
|
spark-env.cmd
|
|
bootstrap.bundle.min.js
|
|
bootstrap.bundle.min.js.map
|
|
bootstrap.min.css
|
|
bootstrap.min.css.map
|
|
jquery-3.4.1.min.js
|
|
d3.min.js
|
|
dagre-d3.min.js
|
|
graphlib-dot.min.js
|
|
sorttable.js
|
|
vis-timeline-graph2d.min.js
|
|
vis-timeline-graph2d.min.css
|
|
dataTables.bootstrap4.1.10.20.min.css
|
|
dataTables.bootstrap4.1.10.20.min.js
|
|
dataTables.rowsGroup.js
|
|
jquery.blockUI.min.js
|
|
jquery.cookies.2.2.0.min.js
|
|
jquery.dataTables.1.10.20.min.css
|
|
jquery.dataTables.1.10.20.min.js
|
|
jquery.mustache.js
|
|
jsonFormatter.min.css
|
|
jsonFormatter.min.js
|
|
.*avsc
|
|
.*txt
|
|
.*json
|
|
.*data
|
|
.*log
|
|
pyspark-coverage-site/
|
|
cloudpickle.py
|
|
heapq3.py
|
|
join.py
|
|
SparkExprTyper.scala
|
|
SparkILoop.scala
|
|
SparkILoopInit.scala
|
|
SparkIMain.scala
|
|
SparkImports.scala
|
|
SparkJLineCompletion.scala
|
|
SparkJLineReader.scala
|
|
SparkMemberHandlers.scala
|
|
SparkReplReporter.scala
|
|
sbt
|
|
sbt-launch-lib.bash
|
|
plugins.sbt
|
|
work
|
|
.*\.q
|
|
.*\.qv
|
|
golden
|
|
test.out/*
|
|
.*iml
|
|
service.properties
|
|
db.lck
|
|
build/*
|
|
dist/*
|
|
.*out
|
|
.*ipr
|
|
.*iws
|
|
logs
|
|
.*scalastyle-output.xml
|
|
.*dependency-reduced-pom.xml
|
|
known_translations
|
|
json_expectation
|
|
app-20180109111548-0000
|
|
app-20161115172038-0000
|
|
app-20161116163331-0000
|
|
application_1516285256255_0012
|
|
application_1553914137147_0018
|
|
application_1555004656427_0144
|
|
stat
|
|
local-1422981759269
|
|
local-1422981780767
|
|
local-1425081759269
|
|
local-1426533911241
|
|
local-1426633911242
|
|
local-1430917381534
|
|
local-1430917381535_1
|
|
local-1430917381535_2
|
|
DESCRIPTION
|
|
NAMESPACE
|
|
test_support/*
|
|
.*Rd
|
|
help/*
|
|
html/*
|
|
INDEX
|
|
.lintr
|
|
gen-java.*
|
|
.*avpr
|
|
.*parquet
|
|
spark-deps-.*
|
|
.*csv
|
|
.*tsv
|
|
.*\.sql
|
|
.Rbuildignore
|
|
META-INF/*
|
|
spark-warehouse
|
|
structured-streaming/*
|
|
kafka-source-initial-offset-version-2.1.0.bin
|
|
kafka-source-initial-offset-future-version.bin
|
|
announce.tmpl
|
|
vote.tmpl
|
|
SessionManager.java
|
|
SessionHandler.java
|
|
GangliaReporter.java
|
|
application_1578436911597_0052
|
|
config.properties
|