ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-11474][SQL] change fetchSize to fetchsize

Browse source 
In DefaultDataSource.scala, it has
override def createRelation(
sqlContext: SQLContext,
parameters: Map[String, String]): BaseRelation
The parameters is CaseInsensitiveMap.
After this line
parameters.foreach(kv => properties.setProperty(kv._1, kv._2))
properties is set to all lower case key/value pairs and fetchSize becomes fetchsize.
However, in compute method in JDBCRDD, it has
val fetchSize = properties.getProperty("fetchSize", "0").toInt
so fetchSize value is always 0 and never gets set correctly.

Author: Huaxin Gao <huaxing@oc0558782468.ibm.com>

Closes #9473 from huaxingao/spark-11474.
This commit is contained in:
Huaxin Gao 2015-11-05 09:41:14 -08:00 committed by Reynold Xin
parent a4b5cefcf1
commit b072ff4d1d
1 changed files with 2 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
View file

@ -347,6 +347,7 @@ private[sql] class JDBCRDD(
/** /**
* Runs the SQL query against the JDBC driver. * Runs the SQL query against the JDBC driver.
*
*/ */
override def compute(thePart: Partition, context: TaskContext): Iterator[InternalRow] = override def compute(thePart: Partition, context: TaskContext): Iterator[InternalRow] =
new Iterator[InternalRow] { new Iterator[InternalRow] {
@ -368,7 +369,7 @@ private[sql] class JDBCRDD(
val sqlText = s"SELECT $columnList FROM $fqTable $myWhereClause" val sqlText = s"SELECT $columnList FROM $fqTable $myWhereClause"
val stmt = conn.prepareStatement(sqlText, val stmt = conn.prepareStatement(sqlText,
ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY) ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)
val fetchSize = properties.getProperty("fetchSize", "0").toInt val fetchSize = properties.getProperty("fetchsize", "0").toInt
stmt.setFetchSize(fetchSize) stmt.setFetchSize(fetchSize)
val rs = stmt.executeQuery() val rs = stmt.executeQuery()