e2d984aa1c
### What changes were proposed in this pull request?
There are currently the R test failures after upgrading `testthat` to 2.0.0, and R version 3.5.2 as of SPARK-23435. This PR targets to fix the tests and make the tests pass. See the explanations and causes below:
```
test_context.R:49: failure: Check masked functions
length(maskedCompletely) not equal to length(namesOfMaskedCompletely).
1/1 mismatches
[1] 6 - 4 == 2
test_context.R:53: failure: Check masked functions
sort(maskedCompletely, na.last = TRUE) not equal to sort(namesOfMaskedCompletely, na.last = TRUE).
5/6 mismatches
x[2]: "endsWith"
y[2]: "filter"
x[3]: "filter"
y[3]: "not"
x[4]: "not"
y[4]: "sample"
x[5]: "sample"
y[5]: NA
x[6]: "startsWith"
y[6]: NA
```
From my cursory look, R base and R's version are mismatched. I fixed accordingly and Jenkins will test it out.
```
test_includePackage.R:31: error: include inside function
package or namespace load failed for ���plyr���:
package ���plyr��� was installed by an R version with different internals; it needs to be reinstalled for use with this R version
Seems it's a package installation issue. Looks like plyr has to be re-installed.
```
From my cursory look, previously installed `plyr` remains and it's not compatible with the new R version. I fixed accordingly and Jenkins will test it out.
```
test_sparkSQL.R:499: warning: SPARK-17811: can create DataFrame containing NA as date and time
Your system is mis-configured: ���/etc/localtime��� is not a symlink
```
Seems a env problem. I suppressed the warnings for now.
```
test_sparkSQL.R:499: warning: SPARK-17811: can create DataFrame containing NA as date and time
It is strongly recommended to set envionment variable TZ to ���America/Los_Angeles��� (or equivalent)
```
Seems a env problem. I suppressed the warnings for now.
```
test_sparkSQL.R:1814: error: string operators
unable to find an inherited method for function ���startsWith��� for signature ���"character"���
1: expect_true(startsWith("Hello World", "Hello")) at /home/jenkins/workspace/SparkPullRequestBuilder2/R/pkg/tests/fulltests/test_sparkSQL.R:1814
2: quasi_label(enquo(object), label)
3: eval_bare(get_expr(quo), get_env(quo))
4: startsWith("Hello World", "Hello")
5: (function (classes, fdef, mtable)
{
methods <- .findInheritedMethods(classes, fdef, mtable)
if (length(methods) == 1L)
return(methods[[1L]])
else if (length(methods) == 0L) {
cnames <- paste0("\"", vapply(classes, as.character, ""), "\"", collapse = ", ")
stop(gettextf("unable to find an inherited method for function %s for signature %s",
sQuote(fdefgeneric), sQuote(cnames)), domain = NA)
}
else stop("Internal error in finding inherited methods; didn't return a unique method",
domain = NA)
})(list("character"), new("nonstandardGenericFunction", .Data = function (x, prefix)
{
standardGeneric("startsWith")
}, generic = structure("startsWith", package = "SparkR"), package = "SparkR", group = list(),
valueClass = character(0), signature = c("x", "prefix"), default = NULL, skeleton = (function (x,
prefix)
stop("invalid call in method dispatch to 'startsWith' (no default method)", domain = NA))(x,
prefix)), <environment>)
6: stop(gettextf("unable to find an inherited method for function %s for signature %s",
sQuote(fdefgeneric), sQuote(cnames)), domain = NA)
```
From my cursory look, R base and R's version are mismatched. I fixed accordingly and Jenkins will test it out.
Also, this PR causes a CRAN check failure as below:
```
* creating vignettes ... ERROR
Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics:
package ���htmltools��� was installed by an R version with different internals; it needs to be reinstalled for use with this R version
```
This PR disables it for now.
### Why are the changes needed?
To unblock other PRs.
### Does this PR introduce any user-facing change?
No. Test only and dev only.
### How was this patch tested?
No. I am going to use Jenkins to test.
Closes #27460 from HyukjinKwon/r-test-failure.
Authored-by: HyukjinKwon <gurwls223@apache.org>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
|
||
|---|---|---|
| .. | ||
| pkg | ||
| .gitignore | ||
| check-cran.sh | ||
| CRAN_RELEASE.md | ||
| create-docs.sh | ||
| create-rd.sh | ||
| DOCUMENTATION.md | ||
| find-r.sh | ||
| install-dev.bat | ||
| install-dev.sh | ||
| install-source-package.sh | ||
| log4j.properties | ||
| README.md | ||
| run-tests.sh | ||
| WINDOWS.md | ||
R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
Installing sparkR
Libraries of sparkR need to be created in $SPARK_HOME/R/lib. This can be done by running the script $SPARK_HOME/R/install-dev.sh.
By default the above script uses the system wide installation of R. However, this can be changed to any user installed location of R by setting the environment variable R_HOME the full path of the base directory where R is installed, before running install-dev.sh script.
Example:
# where /home/username/R is where R is installed and /home/username/R/bin contains the files R and RScript
export R_HOME=/home/username/R
./install-dev.sh
SparkR development
Build Spark
Build Spark with Maven and include the -Psparkr profile to build the R package. For example to use the default Hadoop versions you can run
./build/mvn -DskipTests -Psparkr package
Running sparkR
You can start using SparkR by launching the SparkR shell with
./bin/sparkR
The sparkR script automatically creates a SparkContext with Spark by default in
local mode. To specify the Spark master of a cluster for the automatically created
SparkContext, you can run
./bin/sparkR --master "local[2]"
To set other options like driver memory, executor memory etc. you can pass in the spark-submit arguments to ./bin/sparkR
Using SparkR from RStudio
If you wish to use SparkR from RStudio, please refer SparkR documentation.
Making changes to SparkR
The instructions for making contributions to Spark also apply to SparkR.
If you only make R file changes (i.e. no Scala changes) then you can just re-install the R package using R/install-dev.sh and test your changes.
Once you have made your changes, please include unit tests for them and run existing unit tests using the R/run-tests.sh script as described below.
Generating documentation
The SparkR documentation (Rd files and HTML files) are not a part of the source repository. To generate them you can run the script R/create-docs.sh. This script uses devtools and knitr to generate the docs and these packages need to be installed on the machine before using the script. Also, you may need to install these prerequisites. See also, R/DOCUMENTATION.md
Examples, Unit tests
SparkR comes with several sample programs in the examples/src/main/r directory.
To run one of them, use ./bin/spark-submit <filename> <args>. For example:
./bin/spark-submit examples/src/main/r/dataframe.R
You can run R unit tests by following the instructions under Running R Tests.
Running on YARN
The ./bin/spark-submit can also be used to submit jobs to YARN clusters. You will need to set YARN conf dir before doing so. For example on CDH you can run
export YARN_CONF_DIR=/etc/hadoop/conf
./bin/spark-submit --master yarn examples/src/main/r/dataframe.R