ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-23514] Use SessionState.newHadoopConf() to propage hadoop configs set in SQLConf.

Browse source 
## What changes were proposed in this pull request?

A few places in `spark-sql` were using `sc.hadoopConfiguration` directly. They should be using `sessionState.newHadoopConf()` to blend in configs that were set through `SQLConf`.

Also, for better UX, for these configs blended in from `SQLConf`, we should consider removing the `spark.hadoop` prefix, so that the settings are recognized whether or not they were specified by the user.

## How was this patch tested?

Tested that AlterTableRecoverPartitions now correctly recognizes settings that are passed in to the FileSystem through SQLConf.

Author: Juliusz Sompolski <julek@databricks.com>

Closes #20679 from juliuszsompolski/SPARK-23514.
This commit is contained in:
Juliusz Sompolski 2018-02-28 08:44:53 -08:00 committed by gatorsmile
parent fab563b9bd
commit 476a7f026b
2 changed files with 6 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
View file

@ -610,10 +610,10 @@ case class AlterTableRecoverPartitionsCommand(
val root = new Path(table.location)
logInfo(s"Recover all the partitions in $root")
val fs = root.getFileSystem(spark.sparkContext.hadoopConfiguration)
val hadoopConf = spark.sessionState.newHadoopConf()
val fs = root.getFileSystem(hadoopConf)
val threshold = spark.conf.get("spark.rdd.parallelListingThreshold", "10").toInt
val hadoopConf = spark.sparkContext.hadoopConfiguration
val pathFilter = getPathFilter(hadoopConf)
val evalPool = ThreadUtils.newForkJoinPool("AlterTableRecoverPartitionsCommand", 8)
@ -697,7 +697,7 @@ case class AlterTableRecoverPartitionsCommand(
pathFilter: PathFilter,
threshold: Int): GenMap[String, PartitionStatistics] = {
if (partitionSpecsAndLocs.length > threshold) {
val hadoopConf = spark.sparkContext.hadoopConfiguration
val hadoopConf = spark.sessionState.newHadoopConf()
val serializableConfiguration = new SerializableConfiguration(hadoopConf)
val serializedPaths = partitionSpecsAndLocs.map(_._2.toString).toArray

5
sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala
View file

@ -518,8 +518,9 @@ private[hive] class TestHiveSparkSession(
// an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with
// ${hive.scratch.dir.permission}. To resolve the permission issue, the simplest way is to
// delete it. Later, it will be re-created with the right permission.
val location = new Path(sc.hadoopConfiguration.get(ConfVars.SCRATCHDIR.varname))
val fs = location.getFileSystem(sc.hadoopConfiguration)
val hadoopConf = sessionState.newHadoopConf()
val location = new Path(hadoopConf.get(ConfVars.SCRATCHDIR.varname))
val fs = location.getFileSystem(hadoopConf)
fs.delete(location, true)
// Some tests corrupt this value on purpose, which breaks the RESET call below.