5cc503f4fe
## What changes were proposed in this pull request? If the R data structure that is being parallelized is larger than `INT_MAX` we use files to transfer data to JVM. The serialization protocol mimics Python pickling. This allows us to simply call `PythonRDD.readRDDFromFile` to create the RDD. I tested this on my MacBook. Following code works with this patch: ```R intMax <- .Machine$integer.max largeVec <- 1:intMax rdd <- SparkR:::parallelize(sc, largeVec, 2) ``` ## How was this patch tested? * [x] Unit tests Author: Hossein <hossein@databricks.com> Closes #15375 from falaki/SPARK-17790. |
||
|---|---|---|
| .. | ||
| inst | ||
| R | ||
| src-native | ||
| tests | ||
| vignettes | ||
| .lintr | ||
| .Rbuildignore | ||
| DESCRIPTION | ||
| NAMESPACE | ||