ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-2965][SQL] Fix HashOuterJoin output nullabilities.

Browse source 
Output attributes of opposite side of `OuterJoin` should be nullable.

Author: Takuya UESHIN <ueshin@happy-camper.st>

Closes #1887 from ueshin/issues/SPARK-2965 and squashes the following commits:

bcb2d37 [Takuya UESHIN] Fix HashOuterJoin output nullabilities.
This commit is contained in:
Takuya UESHIN 2014-08-11 20:15:01 -07:00 committed by Michael Armbrust
parent 647aeba3a9
commit c9c89c31b6
1 changed files with 12 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala
View file

@ -168,7 +168,18 @@ case class HashOuterJoin(
override def requiredChildDistribution =
ClusteredDistribution(leftKeys) :: ClusteredDistribution(rightKeys) :: Nil
def output = left.output ++ right.output
override def output = {
joinType match {
case LeftOuter =>
left.output ++ right.output.map(_.withNullability(true))
case RightOuter =>
left.output.map(_.withNullability(true)) ++ right.output
case FullOuter =>
left.output.map(_.withNullability(true)) ++ right.output.map(_.withNullability(true))
case x =>
throw new Exception(s"HashOuterJoin should not take $x as the JoinType")
}
}
// TODO we need to rewrite all of the iterators with our own implementation instead of the Scala
// iterator for performance purpose.