spark-instrumented-optimizer/sql/catalyst/src/main
Xiao Li 9a6ac7226f [SPARK-19601][SQL] Fix CollapseRepartition rule to preserve shuffle-enabled Repartition
### What changes were proposed in this pull request?

Observed by felixcheung  in https://github.com/apache/spark/pull/16739, when users use the shuffle-enabled `repartition` API, they expect the partition they got should be the exact number they provided, even if they call shuffle-disabled `coalesce` later.

Currently, `CollapseRepartition` rule does not consider whether shuffle is enabled or not. Thus, we got the following unexpected result.

```Scala
    val df = spark.range(0, 10000, 1, 5)
    val df2 = df.repartition(10)
    assert(df2.coalesce(13).rdd.getNumPartitions == 5)
    assert(df2.coalesce(7).rdd.getNumPartitions == 5)
    assert(df2.coalesce(3).rdd.getNumPartitions == 3)
```

This PR is to fix the issue. We preserve shuffle-enabled Repartition.

### How was this patch tested?
Added a test case

Author: Xiao Li <gatorsmile@gmail.com>

Closes #16933 from gatorsmile/CollapseRepartition.
2017-03-08 09:36:01 -08:00
..
antlr4/org/apache/spark/sql/catalyst/parser [SPARK-17078][SQL] Show stats when explain 2017-02-24 10:24:59 -08:00
java/org/apache/spark/sql [MINOR][BUILD] Fix lint-java breaks in Java 2017-02-27 08:44:26 +00:00
scala/org/apache/spark/sql [SPARK-19601][SQL] Fix CollapseRepartition rule to preserve shuffle-enabled Repartition 2017-03-08 09:36:01 -08:00