ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-16302][SQL] Set the right number of partitions for reading data from a local collection.

Browse source 
follow #13137 This pr sets the right number of partitions when reading data from a local collection.
Query 'val df = Seq((1, 2)).toDF("key", "value").count' always use defaultParallelism tasks. So it causes run many empty or small tasks.

Manually tested and checked.

Author: Lianhui Wang <lianhuiwang09@gmail.com>

Closes #13979 from lianhuiwang/localTable-Parallel.
This commit is contained in:
Lianhui Wang 2016-09-01 17:08:33 -07:00 committed by Josh Rosen
parent 5bea8757cc
commit 06e33985c6
1 changed files with 4 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScanExec.scala
View file

@ -42,7 +42,10 @@ case class LocalTableScanExec(
}
}
private lazy val rdd = sqlContext.sparkContext.parallelize(unsafeRows)
private lazy val numParallelism: Int = math.min(math.max(unsafeRows.length, 1),
sqlContext.sparkContext.defaultParallelism)
private lazy val rdd = sqlContext.sparkContext.parallelize(unsafeRows, numParallelism)
protected override def doExecute(): RDD[InternalRow] = {
val numOutputRows = longMetric("numOutputRows")