spark-instrumented-optimizer/R/pkg
Xiao Li 9a6ac7226f [SPARK-19601][SQL] Fix CollapseRepartition rule to preserve shuffle-enabled Repartition
### What changes were proposed in this pull request?

Observed by felixcheung  in https://github.com/apache/spark/pull/16739, when users use the shuffle-enabled `repartition` API, they expect the partition they got should be the exact number they provided, even if they call shuffle-disabled `coalesce` later.

Currently, `CollapseRepartition` rule does not consider whether shuffle is enabled or not. Thus, we got the following unexpected result.

```Scala
    val df = spark.range(0, 10000, 1, 5)
    val df2 = df.repartition(10)
    assert(df2.coalesce(13).rdd.getNumPartitions == 5)
    assert(df2.coalesce(7).rdd.getNumPartitions == 5)
    assert(df2.coalesce(3).rdd.getNumPartitions == 3)
```

This PR is to fix the issue. We preserve shuffle-enabled Repartition.

### How was this patch tested?
Added a test case

Author: Xiao Li <gatorsmile@gmail.com>

Closes #16933 from gatorsmile/CollapseRepartition.
2017-03-08 09:36:01 -08:00
..
inst [SPARK-19601][SQL] Fix CollapseRepartition rule to preserve shuffle-enabled Repartition 2017-03-08 09:36:01 -08:00
R [SPARK-19818][SPARKR] rbind should check for name consistency of input data frames 2017-03-06 21:55:11 -08:00
src-native [SPARK-6811] Copy SparkR lib in make-distribution.sh 2015-05-23 00:04:01 -07:00
tests [SPARK-19387][SPARKR] Tests do not run with SparkR source package in CRAN check 2017-02-14 13:51:27 -08:00
vignettes [SPARK-19460][SPARKR] Update dataset used in R documentation, examples to reduce warning noise and confusions 2017-02-28 22:31:35 -08:00
.lintr [SPARK-12327][SPARKR] fix code for lintr warning for commented code 2016-01-03 20:53:35 +05:30
.Rbuildignore [SPARK-18590][SPARKR] build R source package when making distribution 2016-12-08 11:29:31 -08:00
DESCRIPTION [SPARK-18862][SPARKR][ML] Split SparkR mllib.R into multiple files 2017-01-08 01:10:36 -08:00
NAMESPACE [SPARK-19795][SPARKR] add column functions to_json, from_json 2017-03-05 12:37:02 -08:00