0cf2f48dfe
### What changes were proposed in this pull request? For issue mentioned in [SPARK-29022](https://issues.apache.org/jira/browse/SPARK-29022) Spark SQL CLI can't use class as serde class in jars add by SQL `ADD JAR`. When we create table with `serde` class contains by jar added by SQL 'ADD JAR'. We can create table with `serde` class construct success since we call `HiveClientImpl.createTable` under `withHiveState` method, it will add `clientLoader.classLoader` to `HiveClientImpl.state.getConf.classLoader`. Jars added by SQL `ADD JAR` will be add to 1. `sparkSession.sharedState.jarClassLoader`. 2. 'HiveClientLoader.clientLoader.classLoader' In Current spark-sql MODE, `HiveClientImpl.state` will use CliSessionState created when initialize SparkSQLCliDriver, When we select data from table, it will check `serde` class, when call method `HiveTableScanExec#addColumnMetadataToConf()` to check for table desc serde class. ``` val deserializer = tableDesc.getDeserializerClass.getConstructor().newInstance() deserializer.initialize(hiveConf, tableDesc.getProperties) ``` `getDeserializer` will use CliSessionState's hiveConf's classLoader in `Spark SQL CLI` mode. But when we call `ADD JAR` in spark, the jar won't be added to `Classloader of CliSessionState' conf `, then `ClassNotFound` error happen. So we reset `CliSessionState conf's classLoader ` to `sharedState.jarClassLoader` when `sharedState.jarClassLoader` has added jar passed by `HIVEAUXJARS` Then when we use `ADD JAR ` to add jar, jar path will be added to CliSessionState's conf's ClassLoader ### Why are the changes needed? Fix bug ### Does this PR introduce any user-facing change? No ### How was this patch tested? ADD UT Closes #25729 from AngersZhuuuu/SPARK-29015. Authored-by: angerszhu <angers.zhu@gmail.com> Signed-off-by: Sean Owen <sean.owen@databricks.com> |
||
---|---|---|
.. | ||
src | ||
v1.2.1 | ||
v2.3.5 | ||
pom.xml |