ODIn/spark-instrumented-optimizer
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[SPARK-17182][SQL] Mark Collect as non-deterministic

Browse source 
## What changes were proposed in this pull request?

This PR marks the abstract class `Collect` as non-deterministic since the results of `CollectList` and `CollectSet` depend on the actual order of input rows.

## How was this patch tested?

Existing test cases should be enough.

Author: Cheng Lian <lian@databricks.com>

Closes #14749 from liancheng/spark-17182-non-deterministic-collect.
This commit is contained in:
Cheng Lian 2016-08-23 09:11:47 +08:00 committed by Wenchen Fan
parent 920806ab27
commit 2cdd92a7cd
1 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
View file

@ -54,6 +54,10 @@ abstract class Collect extends ImperativeAggregate {
override def inputAggBufferAttributes: Seq[AttributeReference] = Nil
// Both `CollectList` and `CollectSet` are non-deterministic since their results depend on the
// actual order of input rows.
override def deterministic: Boolean = false
protected[this] val buffer: Growable[Any] with Iterable[Any]
override def initialize(b: MutableRow): Unit = {