[SPARK-8020] [SQL] Spark SQL conf in spark-defaults.conf make metadataHive get constructed too early
https://issues.apache.org/jira/browse/SPARK-8020
Author: Yin Huai <yhuai@databricks.com>
Closes #6571 from yhuai/SPARK-8020-1 and squashes the following commits:
0398f5b [Yin Huai] First populate the SQLConf and then construct executionHive and metadataHive.
(cherry picked from commit 7b7f7b6c6f
)
Signed-off-by: Yin Huai <yhuai@databricks.com>
This commit is contained in:
parent
9d6475b93d
commit
4940630f56
|
@ -182,10 +182,29 @@ class SQLContext(@transient val sparkContext: SparkContext)
|
||||||
conf.dialect
|
conf.dialect
|
||||||
}
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
// We extract spark sql settings from SparkContext's conf and put them to
|
||||||
|
// Spark SQL's conf.
|
||||||
|
// First, we populate the SQLConf (conf). So, we can make sure that other values using
|
||||||
|
// those settings in their construction can get the correct settings.
|
||||||
|
// For example, metadataHive in HiveContext may need both spark.sql.hive.metastore.version
|
||||||
|
// and spark.sql.hive.metastore.jars to get correctly constructed.
|
||||||
|
val properties = new Properties
|
||||||
sparkContext.getConf.getAll.foreach {
|
sparkContext.getConf.getAll.foreach {
|
||||||
case (key, value) if key.startsWith("spark.sql") => setConf(key, value)
|
case (key, value) if key.startsWith("spark.sql") => properties.setProperty(key, value)
|
||||||
case _ =>
|
case _ =>
|
||||||
}
|
}
|
||||||
|
// We directly put those settings to conf to avoid of calling setConf, which may have
|
||||||
|
// side-effects. For example, in HiveContext, setConf may cause executionHive and metadataHive
|
||||||
|
// get constructed. If we call setConf directly, the constructed metadataHive may have
|
||||||
|
// wrong settings, or the construction may fail.
|
||||||
|
conf.setConf(properties)
|
||||||
|
// After we have populated SQLConf, we call setConf to populate other confs in the subclass
|
||||||
|
// (e.g. hiveconf in HiveContext).
|
||||||
|
properties.foreach {
|
||||||
|
case (key, value) => setConf(key, value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@transient
|
@transient
|
||||||
protected[sql] val cacheManager = new CacheManager(this)
|
protected[sql] val cacheManager = new CacheManager(this)
|
||||||
|
|
Loading…
Reference in a new issue